2024-03-11T22:36:46.724 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-11T22:36:46.725 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name dis-2024-03-11_22:26:48-rbd-wip-dis-testing-distro-default-smithi --archive /home/teuthworker/archive/dis-2024-03-11_22:26:48-rbd-wip-dis-testing-distro-default-smithi/7592296 --description rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum} --owner scheduled_dis@teuthology -- /home/teuthworker/archive/dis-2024-03-11_22:26:48-rbd-wip-dis-testing-distro-default-smithi/7592296/orig.config.yaml 2024-03-11T22:36:46.773 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-11T22:36:46.859 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dis-2024-03-11_22:26:48-rbd-wip-dis-testing-distro-default-smithi/7592296 branch: wip-dis-testing description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum} email: idryomov@gmail.com first_in_suite: false job_id: '7592296' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: dis-2024-03-11_22:26:48-rbd-wip-dis-testing-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-dis-testing ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: simple client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: b117a47aa0ad81d207fdd9ba852068f2f262fea8 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: b117a47aa0ad81d207fdd9ba852068f2f262fea8 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-dis-testing-qa sha1: a4679433626e24f222f64f757c6ab31760167603 owner: scheduled_dis@teuthology priority: 70 repo: https://github.com/ceph/ceph-ci roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 5633 sha1: b117a47aa0ad81d207fdd9ba852068f2f262fea8 sleep_before_teardown: 0 subset: 81/99 suite: rbd suite_branch: wip-dis-testing-qa suite_path: /home/teuthworker/src/github.com_idryomov_ceph_a4679433626e24f222f64f757c6ab31760167603/qa suite_relpath: qa suite_repo: https://github.com/idryomov/ceph suite_sha1: a4679433626e24f222f64f757c6ab31760167603 targets: smithi078.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBALYIufoA3dz1qKE8kADQ7x1+cDKs+cZ3+eoCp4Ma9/PkJRs0d6nG11XqINqKSmIeLmh1eTH34m/OIJCkaVKVzE= smithi132.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPNf1BzILs3QC7X/vb7hhud5sQ79Ov/m2HPKnz8nSwAUei2BPhVJ2BlnaQ0g4BlXne1kwcu0uHG9hKemdzGeUQw= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_POOL_MODE: image RBD_IMAGE_FEATURES: layering RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_MODE: snapshot RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-11_22:26:48 tube: smithi user: dis verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-11T22:36:46.860 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_idryomov_ceph_a4679433626e24f222f64f757c6ab31760167603/qa; will attempt to use it 2024-03-11T22:36:46.878 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_idryomov_ceph_a4679433626e24f222f64f757c6ab31760167603/qa/tasks 2024-03-11T22:36:46.878 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-11T22:36:46.880 INFO:teuthology.task.internal:Checking packages... 2024-03-11T22:36:46.902 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'b117a47aa0ad81d207fdd9ba852068f2f262fea8' 2024-03-11T22:36:46.902 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-11T22:36:46.902 INFO:teuthology.packaging:ref: None 2024-03-11T22:36:46.902 INFO:teuthology.packaging:tag: None 2024-03-11T22:36:46.902 INFO:teuthology.packaging:branch: wip-dis-testing 2024-03-11T22:36:46.902 INFO:teuthology.packaging:sha1: b117a47aa0ad81d207fdd9ba852068f2f262fea8 2024-03-11T22:36:46.902 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-dis-testing 2024-03-11T22:36:47.055 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1979.gb117a47a 2024-03-11T22:36:47.056 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-11T22:36:47.062 INFO:teuthology.task.internal:no buildpackages task found 2024-03-11T22:36:47.063 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-11T22:36:47.161 INFO:teuthology.task.internal:Saving configuration 2024-03-11T22:36:47.179 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-11T22:36:47.184 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T22:36:47.213 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi078.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2024-03-11_22:26:48-rbd-wip-dis-testing-distro-default-smithi/7592296', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 22:31:53.751791', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBALYIufoA3dz1qKE8kADQ7x1+cDKs+cZ3+eoCp4Ma9/PkJRs0d6nG11XqINqKSmIeLmh1eTH34m/OIJCkaVKVzE='} 2024-03-11T22:36:47.236 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi132.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2024-03-11_22:26:48-rbd-wip-dis-testing-distro-default-smithi/7592296', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 22:31:53.749756', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPNf1BzILs3QC7X/vb7hhud5sQ79Ov/m2HPKnz8nSwAUei2BPhVJ2BlnaQ0g4BlXne1kwcu0uHG9hKemdzGeUQw='} 2024-03-11T22:36:47.236 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-11T22:36:47.242 INFO:teuthology.task.internal:roles: ubuntu@smithi078.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2024-03-11T22:36:47.242 INFO:teuthology.task.internal:roles: ubuntu@smithi132.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2024-03-11T22:36:47.242 INFO:teuthology.run_tasks:Running task console_log... 2024-03-11T22:36:47.304 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7289f54ee0>, signals=[15]) 2024-03-11T22:36:47.305 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-11T22:36:47.310 INFO:teuthology.task.internal:Opening connections... 2024-03-11T22:36:47.311 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi078.front.sepia.ceph.com 2024-03-11T22:36:47.312 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:36:47.402 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi132.front.sepia.ceph.com 2024-03-11T22:36:47.403 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:36:47.492 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-11T22:36:47.499 DEBUG:teuthology.orchestra.run.smithi078:> uname -m 2024-03-11T22:36:47.516 INFO:teuthology.orchestra.run.smithi078.stdout:x86_64 2024-03-11T22:36:47.517 DEBUG:teuthology.orchestra.run.smithi078:> cat /etc/os-release 2024-03-11T22:36:47.576 INFO:teuthology.orchestra.run.smithi078.stdout:NAME="CentOS Stream" 2024-03-11T22:36:47.576 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION="9" 2024-03-11T22:36:47.576 INFO:teuthology.orchestra.run.smithi078.stdout:ID="centos" 2024-03-11T22:36:47.576 INFO:teuthology.orchestra.run.smithi078.stdout:ID_LIKE="rhel fedora" 2024-03-11T22:36:47.576 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_ID="9" 2024-03-11T22:36:47.576 INFO:teuthology.orchestra.run.smithi078.stdout:PLATFORM_ID="platform:el9" 2024-03-11T22:36:47.576 INFO:teuthology.orchestra.run.smithi078.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T22:36:47.576 INFO:teuthology.orchestra.run.smithi078.stdout:ANSI_COLOR="0;31" 2024-03-11T22:36:47.577 INFO:teuthology.orchestra.run.smithi078.stdout:LOGO="fedora-logo-icon" 2024-03-11T22:36:47.577 INFO:teuthology.orchestra.run.smithi078.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T22:36:47.577 INFO:teuthology.orchestra.run.smithi078.stdout:HOME_URL="https://centos.org/" 2024-03-11T22:36:47.577 INFO:teuthology.orchestra.run.smithi078.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T22:36:47.577 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T22:36:47.577 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T22:36:47.578 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server 2024-03-11T22:36:47.599 DEBUG:teuthology.orchestra.run.smithi132:> uname -m 2024-03-11T22:36:47.616 INFO:teuthology.orchestra.run.smithi132.stdout:x86_64 2024-03-11T22:36:47.616 DEBUG:teuthology.orchestra.run.smithi132:> cat /etc/os-release 2024-03-11T22:36:47.672 INFO:teuthology.orchestra.run.smithi132.stdout:NAME="CentOS Stream" 2024-03-11T22:36:47.672 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION="9" 2024-03-11T22:36:47.672 INFO:teuthology.orchestra.run.smithi132.stdout:ID="centos" 2024-03-11T22:36:47.672 INFO:teuthology.orchestra.run.smithi132.stdout:ID_LIKE="rhel fedora" 2024-03-11T22:36:47.672 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION_ID="9" 2024-03-11T22:36:47.672 INFO:teuthology.orchestra.run.smithi132.stdout:PLATFORM_ID="platform:el9" 2024-03-11T22:36:47.672 INFO:teuthology.orchestra.run.smithi132.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T22:36:47.672 INFO:teuthology.orchestra.run.smithi132.stdout:ANSI_COLOR="0;31" 2024-03-11T22:36:47.672 INFO:teuthology.orchestra.run.smithi132.stdout:LOGO="fedora-logo-icon" 2024-03-11T22:36:47.672 INFO:teuthology.orchestra.run.smithi132.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T22:36:47.672 INFO:teuthology.orchestra.run.smithi132.stdout:HOME_URL="https://centos.org/" 2024-03-11T22:36:47.672 INFO:teuthology.orchestra.run.smithi132.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T22:36:47.672 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T22:36:47.672 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T22:36:47.673 INFO:teuthology.lock.ops:Updating smithi132.front.sepia.ceph.com on lock server 2024-03-11T22:36:47.690 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-11T22:36:47.701 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-11T22:36:47.790 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-11T22:36:47.790 DEBUG:teuthology.orchestra.run.smithi078:> test '!' -e /home/ubuntu/cephtest 2024-03-11T22:36:47.793 DEBUG:teuthology.orchestra.run.smithi132:> test '!' -e /home/ubuntu/cephtest 2024-03-11T22:36:47.808 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-11T22:36:47.813 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-11T22:36:47.813 DEBUG:teuthology.orchestra.run.smithi078:> test -z $(ls -A /var/lib/ceph) 2024-03-11T22:36:47.849 DEBUG:teuthology.orchestra.run.smithi132:> test -z $(ls -A /var/lib/ceph) 2024-03-11T22:36:47.962 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-11T22:36:48.015 INFO:teuthology.run_tasks:Running task kernel... 2024-03-11T22:36:48.029 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-11T22:36:48.030 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-11T22:36:48.030 DEBUG:teuthology.orchestra.run.smithi078:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T22:36:48.031 DEBUG:teuthology.orchestra.run.smithi132:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T22:36:48.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:36:48.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:36:48.047 DEBUG:teuthology.orchestra.run.smithi132:> uname -r 2024-03-11T22:36:48.048 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2024-03-11T22:36:48.104 INFO:teuthology.orchestra.run.smithi078.stdout:5.14.0-361.el9.x86_64 2024-03-11T22:36:48.104 INFO:teuthology.task.kernel:Running kernel on smithi078: 5.14.0-361.el9.x86_64 2024-03-11T22:36:48.104 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum install -y kernel 2024-03-11T22:36:48.105 INFO:teuthology.orchestra.run.smithi132.stdout:5.14.0-361.el9.x86_64 2024-03-11T22:36:48.105 INFO:teuthology.task.kernel:Running kernel on smithi132: 5.14.0-361.el9.x86_64 2024-03-11T22:36:48.105 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum install -y kernel 2024-03-11T22:36:50.225 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-11T22:36:50.291 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-11T22:36:51.700 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS-9 - BaseOS 5.6 MB/s | 2.0 MB 00:00 2024-03-11T22:36:51.900 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-11T22:36:54.816 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - BaseOS 2.9 MB/s | 8.0 MB 00:02 2024-03-11T22:36:57.190 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - BaseOS 1.7 MB/s | 8.0 MB 00:04 2024-03-11T22:36:59.099 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-11T22:37:02.699 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - AppStream 2.8 MB/s | 19 MB 00:06 2024-03-11T22:37:04.600 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - CRB 3.1 MB/s | 6.1 MB 00:01 2024-03-11T22:37:06.067 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 15 kB 00:00 2024-03-11T22:37:07.118 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - CRB 6.9 MB/s | 6.1 MB 00:00 2024-03-11T22:37:08.709 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 15 kB 00:00 2024-03-11T22:37:09.968 INFO:teuthology.orchestra.run.smithi132.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-03-11T22:37:11.702 INFO:teuthology.orchestra.run.smithi078.stdout:Extra Packages for Enterprise Linux 3.7 MB/s | 21 MB 00:05 2024-03-11T22:37:14.943 INFO:teuthology.orchestra.run.smithi132.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-11T22:37:16.852 INFO:teuthology.orchestra.run.smithi078.stdout:lab-extras 5.6 kB/s | 1.7 kB 00:00 2024-03-11T22:37:17.142 INFO:teuthology.orchestra.run.smithi132.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T22:37:17.142 INFO:teuthology.orchestra.run.smithi132.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T22:37:17.230 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-11T22:37:17.237 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-11T22:37:17.237 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2024-03-11T22:37:17.237 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-11T22:37:17.237 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-11T22:37:17.237 INFO:teuthology.orchestra.run.smithi132.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T22:37:17.237 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-03-11T22:37:17.237 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T22:37:17.237 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T22:37:17.237 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T22:37:17.237 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:37:17.237 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-11T22:37:17.237 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-11T22:37:17.237 INFO:teuthology.orchestra.run.smithi132.stdout:Install 4 Packages 2024-03-11T22:37:17.238 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:37:17.238 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 100 M 2024-03-11T22:37:17.238 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 123 M 2024-03-11T22:37:17.238 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-11T22:37:18.053 INFO:teuthology.orchestra.run.smithi132.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 8.0 MB/s | 6.0 MB 00:00 2024-03-11T22:37:18.320 INFO:teuthology.orchestra.run.smithi132.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 20 MB/s | 21 MB 00:01 2024-03-11T22:37:19.008 INFO:teuthology.orchestra.run.smithi078.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T22:37:19.008 INFO:teuthology.orchestra.run.smithi078.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T22:37:19.100 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-03-11T22:37:19.107 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-11T22:37:19.107 INFO:teuthology.orchestra.run.smithi078.stdout: Package Architecture Version Repository Size 2024-03-11T22:37:19.107 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-11T22:37:19.107 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-03-11T22:37:19.107 INFO:teuthology.orchestra.run.smithi078.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T22:37:19.107 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-03-11T22:37:19.108 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T22:37:19.108 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T22:37:19.108 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T22:37:19.108 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:37:19.108 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-03-11T22:37:19.108 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-11T22:37:19.108 INFO:teuthology.orchestra.run.smithi078.stdout:Install 4 Packages 2024-03-11T22:37:19.108 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:37:19.109 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 100 M 2024-03-11T22:37:19.109 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 123 M 2024-03-11T22:37:19.109 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-03-11T22:37:21.062 INFO:teuthology.orchestra.run.smithi132.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 10 MB/s | 39 MB 00:03 2024-03-11T22:37:21.379 INFO:teuthology.orchestra.run.smithi132.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 10 MB/s | 33 MB 00:03 2024-03-11T22:37:21.379 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:37:21.379 INFO:teuthology.orchestra.run.smithi132.stdout:Total 24 MB/s | 100 MB 00:04 2024-03-11T22:37:21.953 INFO:teuthology.orchestra.run.smithi078.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 2.2 MB/s | 6.0 MB 00:02 2024-03-11T22:37:22.266 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-11T22:37:22.637 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-11T22:37:22.638 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-11T22:37:24.004 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-11T22:37:24.005 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-11T22:37:24.511 INFO:teuthology.orchestra.run.smithi078.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 4.0 MB/s | 21 MB 00:05 2024-03-11T22:37:27.000 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-11T22:37:27.655 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T22:37:27.770 INFO:teuthology.orchestra.run.smithi078.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 4.6 MB/s | 39 MB 00:08 2024-03-11T22:37:27.789 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:37:29.553 INFO:teuthology.orchestra.run.smithi078.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 4.4 MB/s | 33 MB 00:07 2024-03-11T22:37:29.553 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:37:29.554 INFO:teuthology.orchestra.run.smithi078.stdout:Total 9.5 MB/s | 100 MB 00:10 2024-03-11T22:37:30.442 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-03-11T22:37:30.566 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:37:30.761 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:37:30.820 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-03-11T22:37:30.820 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-03-11T22:37:32.152 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-03-11T22:37:32.153 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-03-11T22:37:35.094 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-03-11T22:37:35.520 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:37:35.645 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:37:35.802 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T22:37:35.930 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:37:38.720 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:37:38.907 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:37:40.401 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:37:43.652 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:37:43.769 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:37:48.577 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:38:09.944 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:38:09.945 INFO:teuthology.orchestra.run.smithi132.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T22:38:09.945 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:38:10.031 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:38:10.475 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:38:10.475 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T22:38:10.475 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:38:10.476 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:38:10.920 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:38:10.920 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:38:10.920 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-11T22:38:10.920 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T22:38:10.920 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T22:38:10.920 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T22:38:10.920 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T22:38:10.920 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:38:10.920 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-11T22:38:11.451 DEBUG:teuthology.orchestra.run.smithi132:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T22:38:11.613 INFO:teuthology.orchestra.run.smithi132.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T22:38:11.613 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T22:38:11.614 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T22:38:11.614 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T22:38:11.614 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T22:38:11.614 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2024-03-11T22:38:11.614 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi132.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T22:38:11.614 DEBUG:teuthology.orchestra.run.smithi132:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T22:38:11.735 INFO:teuthology.orchestra.run.smithi132.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T22:38:11.735 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T22:38:11.735 DEBUG:teuthology.orchestra.run.smithi132:> sudo rpm -qi grub2-tools 2024-03-11T22:38:11.769 INFO:teuthology.orchestra.run.smithi132.stdout:Name : grub2-tools 2024-03-11T22:38:11.769 INFO:teuthology.orchestra.run.smithi132.stdout:Epoch : 1 2024-03-11T22:38:11.769 INFO:teuthology.orchestra.run.smithi132.stdout:Version : 2.06 2024-03-11T22:38:11.769 INFO:teuthology.orchestra.run.smithi132.stdout:Release : 68.el9 2024-03-11T22:38:11.769 INFO:teuthology.orchestra.run.smithi132.stdout:Architecture: x86_64 2024-03-11T22:38:11.769 INFO:teuthology.orchestra.run.smithi132.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T22:38:11.769 INFO:teuthology.orchestra.run.smithi132.stdout:Group : Unspecified 2024-03-11T22:38:11.770 INFO:teuthology.orchestra.run.smithi132.stdout:Size : 8264254 2024-03-11T22:38:11.770 INFO:teuthology.orchestra.run.smithi132.stdout:License : GPLv3+ 2024-03-11T22:38:11.770 INFO:teuthology.orchestra.run.smithi132.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T22:38:11.770 INFO:teuthology.orchestra.run.smithi132.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T22:38:11.770 INFO:teuthology.orchestra.run.smithi132.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T22:38:11.770 INFO:teuthology.orchestra.run.smithi132.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T22:38:11.770 INFO:teuthology.orchestra.run.smithi132.stdout:Packager : builder@centos.org 2024-03-11T22:38:11.770 INFO:teuthology.orchestra.run.smithi132.stdout:Vendor : CentOS 2024-03-11T22:38:11.770 INFO:teuthology.orchestra.run.smithi132.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T22:38:11.770 INFO:teuthology.orchestra.run.smithi132.stdout:Summary : Support tools for GRUB. 2024-03-11T22:38:11.770 INFO:teuthology.orchestra.run.smithi132.stdout:Description : 2024-03-11T22:38:11.770 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:38:11.770 INFO:teuthology.orchestra.run.smithi132.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T22:38:11.771 INFO:teuthology.orchestra.run.smithi132.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T22:38:11.771 INFO:teuthology.orchestra.run.smithi132.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T22:38:11.771 INFO:teuthology.orchestra.run.smithi132.stdout:hardware devices. 2024-03-11T22:38:11.771 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:38:11.771 INFO:teuthology.orchestra.run.smithi132.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T22:38:11.771 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T22:38:11.772 INFO:teuthology.task.kernel:Updating grub on smithi132 to boot 5.14.0-427.el9.x86_64 2024-03-11T22:38:11.772 DEBUG:teuthology.orchestra.run.smithi132:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T22:38:12.373 INFO:teuthology.orchestra.run.smithi132.stderr:Generating grub configuration file ... 2024-03-11T22:38:13.706 INFO:teuthology.orchestra.run.smithi132.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T22:38:13.742 INFO:teuthology.orchestra.run.smithi132.stderr:done 2024-03-11T22:38:13.744 DEBUG:teuthology.orchestra.run.smithi132:> mktemp 2024-03-11T22:38:13.759 INFO:teuthology.orchestra.run.smithi132.stdout:/tmp/tmp.j5m9lnDS4H 2024-03-11T22:38:13.759 DEBUG:teuthology.orchestra.run.smithi132:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.j5m9lnDS4H 2024-03-11T22:38:13.827 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 0666 /tmp/tmp.j5m9lnDS4H 2024-03-11T22:38:13.973 DEBUG:teuthology.orchestra.remote:smithi132:/tmp/tmp.j5m9lnDS4H is 6KB 2024-03-11T22:38:14.022 DEBUG:teuthology.orchestra.run.smithi132:> rm -fr /tmp/tmp.j5m9lnDS4H 2024-03-11T22:38:14.036 DEBUG:teuthology.orchestra.run.smithi132:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T22:38:14.117 INFO:teuthology.orchestra.run.smithi132.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T22:38:14.117 INFO:teuthology.orchestra.run.smithi132.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T22:38:14.117 INFO:teuthology.orchestra.run.smithi132.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T22:38:14.117 INFO:teuthology.orchestra.run.smithi132.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T22:38:14.118 DEBUG:teuthology.orchestra.run.smithi132:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T22:38:14.269 DEBUG:teuthology.orchestra.run.smithi132:> sudo shutdown -r now 2024-03-11T22:38:17.792 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:38:17.792 INFO:teuthology.orchestra.run.smithi078.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T22:38:17.792 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:38:17.877 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:38:18.381 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:38:18.381 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T22:38:18.381 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:38:18.381 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:38:18.802 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:38:18.802 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:38:18.803 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-03-11T22:38:18.803 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T22:38:18.803 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T22:38:18.803 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T22:38:18.803 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T22:38:18.803 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:38:18.803 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-03-11T22:38:19.300 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T22:38:19.451 INFO:teuthology.orchestra.run.smithi078.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T22:38:19.451 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T22:38:19.451 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T22:38:19.451 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T22:38:19.451 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T22:38:19.451 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2024-03-11T22:38:19.451 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi078.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T22:38:19.452 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T22:38:19.572 INFO:teuthology.orchestra.run.smithi078.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T22:38:19.572 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T22:38:19.572 DEBUG:teuthology.orchestra.run.smithi078:> sudo rpm -qi grub2-tools 2024-03-11T22:38:19.614 INFO:teuthology.orchestra.run.smithi078.stdout:Name : grub2-tools 2024-03-11T22:38:19.614 INFO:teuthology.orchestra.run.smithi078.stdout:Epoch : 1 2024-03-11T22:38:19.614 INFO:teuthology.orchestra.run.smithi078.stdout:Version : 2.06 2024-03-11T22:38:19.614 INFO:teuthology.orchestra.run.smithi078.stdout:Release : 68.el9 2024-03-11T22:38:19.614 INFO:teuthology.orchestra.run.smithi078.stdout:Architecture: x86_64 2024-03-11T22:38:19.614 INFO:teuthology.orchestra.run.smithi078.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T22:38:19.614 INFO:teuthology.orchestra.run.smithi078.stdout:Group : Unspecified 2024-03-11T22:38:19.614 INFO:teuthology.orchestra.run.smithi078.stdout:Size : 8264254 2024-03-11T22:38:19.614 INFO:teuthology.orchestra.run.smithi078.stdout:License : GPLv3+ 2024-03-11T22:38:19.614 INFO:teuthology.orchestra.run.smithi078.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T22:38:19.615 INFO:teuthology.orchestra.run.smithi078.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T22:38:19.615 INFO:teuthology.orchestra.run.smithi078.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T22:38:19.615 INFO:teuthology.orchestra.run.smithi078.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T22:38:19.615 INFO:teuthology.orchestra.run.smithi078.stdout:Packager : builder@centos.org 2024-03-11T22:38:19.615 INFO:teuthology.orchestra.run.smithi078.stdout:Vendor : CentOS 2024-03-11T22:38:19.615 INFO:teuthology.orchestra.run.smithi078.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T22:38:19.615 INFO:teuthology.orchestra.run.smithi078.stdout:Summary : Support tools for GRUB. 2024-03-11T22:38:19.615 INFO:teuthology.orchestra.run.smithi078.stdout:Description : 2024-03-11T22:38:19.615 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:38:19.615 INFO:teuthology.orchestra.run.smithi078.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T22:38:19.615 INFO:teuthology.orchestra.run.smithi078.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T22:38:19.615 INFO:teuthology.orchestra.run.smithi078.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T22:38:19.615 INFO:teuthology.orchestra.run.smithi078.stdout:hardware devices. 2024-03-11T22:38:19.616 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:38:19.616 INFO:teuthology.orchestra.run.smithi078.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T22:38:19.617 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T22:38:19.617 INFO:teuthology.task.kernel:Updating grub on smithi078 to boot 5.14.0-427.el9.x86_64 2024-03-11T22:38:19.617 DEBUG:teuthology.orchestra.run.smithi078:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T22:38:20.247 INFO:teuthology.orchestra.run.smithi078.stderr:Generating grub configuration file ... 2024-03-11T22:38:21.622 INFO:teuthology.orchestra.run.smithi078.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T22:38:21.659 INFO:teuthology.orchestra.run.smithi078.stderr:done 2024-03-11T22:38:21.661 DEBUG:teuthology.orchestra.run.smithi078:> mktemp 2024-03-11T22:38:21.676 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/tmp.cv6xEUFkqA 2024-03-11T22:38:21.676 DEBUG:teuthology.orchestra.run.smithi078:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.cv6xEUFkqA 2024-03-11T22:38:21.744 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0666 /tmp/tmp.cv6xEUFkqA 2024-03-11T22:38:21.881 DEBUG:teuthology.orchestra.remote:smithi078:/tmp/tmp.cv6xEUFkqA is 6KB 2024-03-11T22:38:21.929 DEBUG:teuthology.orchestra.run.smithi078:> rm -fr /tmp/tmp.cv6xEUFkqA 2024-03-11T22:38:21.943 DEBUG:teuthology.orchestra.run.smithi078:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T22:38:22.008 INFO:teuthology.orchestra.run.smithi078.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T22:38:22.008 INFO:teuthology.orchestra.run.smithi078.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T22:38:22.008 INFO:teuthology.orchestra.run.smithi078.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T22:38:22.008 INFO:teuthology.orchestra.run.smithi078.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T22:38:22.009 DEBUG:teuthology.orchestra.run.smithi078:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T22:38:22.159 DEBUG:teuthology.orchestra.run.smithi078:> sudo shutdown -r now 2024-03-11T22:38:44.294 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2024-03-11T22:38:44.294 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-03-11T22:38:44.295 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:38:52.170 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2024-03-11T22:38:52.170 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-03-11T22:38:52.171 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:39:44.340 DEBUG:teuthology.orchestra.remote:timed out 2024-03-11T22:39:52.175 DEBUG:teuthology.orchestra.remote:timed out 2024-03-11T22:39:53.343 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-03-11T22:39:53.343 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:39:53.721 DEBUG:teuthology.orchestra.run.smithi132:> true 2024-03-11T22:39:54.140 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-03-11T22:39:54.140 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-11T22:39:54.140 DEBUG:teuthology.orchestra.run.smithi132:> uname -r 2024-03-11T22:39:54.153 INFO:teuthology.orchestra.run.smithi132.stdout:5.14.0-427.el9.x86_64 2024-03-11T22:39:54.153 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T22:39:54.154 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T22:39:54.154 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T22:39:55.155 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2024-03-11T22:39:55.155 DEBUG:teuthology.orchestra.run.smithi132:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T22:39:55.241 INFO:teuthology.orchestra.run.smithi132.stdout:ttyS1 2024-03-11T22:39:55.283 DEBUG:teuthology.parallel:result is None 2024-03-11T22:40:01.175 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-03-11T22:40:01.176 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:40:04.591 DEBUG:teuthology.orchestra.run.smithi078:> true 2024-03-11T22:40:05.006 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-03-11T22:40:05.007 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-11T22:40:05.007 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2024-03-11T22:40:05.020 INFO:teuthology.orchestra.run.smithi078.stdout:5.14.0-427.el9.x86_64 2024-03-11T22:40:05.021 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T22:40:05.021 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T22:40:05.021 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T22:40:06.022 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2024-03-11T22:40:06.022 DEBUG:teuthology.orchestra.run.smithi078:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T22:40:06.103 INFO:teuthology.orchestra.run.smithi078.stdout:ttyS1 2024-03-11T22:40:06.152 DEBUG:teuthology.parallel:result is None 2024-03-11T22:40:06.152 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-11T22:40:06.159 INFO:teuthology.task.internal:Creating test directory... 2024-03-11T22:40:06.160 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T22:40:06.162 DEBUG:teuthology.orchestra.run.smithi132:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T22:40:06.180 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-11T22:40:06.188 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-11T22:40:06.197 INFO:teuthology.task.internal:Creating archive directory... 2024-03-11T22:40:06.198 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T22:40:06.217 DEBUG:teuthology.orchestra.run.smithi132:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T22:40:06.331 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-11T22:40:06.338 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-11T22:40:06.338 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T22:40:06.340 DEBUG:teuthology.orchestra.run.smithi132:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T22:40:06.373 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T22:40:06.381 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T22:40:06.386 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T22:40:06.396 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T22:40:06.398 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-11T22:40:06.407 INFO:teuthology.task.internal:Configuring sudo... 2024-03-11T22:40:06.407 DEBUG:teuthology.orchestra.run.smithi078:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T22:40:06.430 DEBUG:teuthology.orchestra.run.smithi132:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T22:40:06.469 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-11T22:40:06.478 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-11T22:40:06.478 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T22:40:06.498 DEBUG:teuthology.orchestra.run.smithi132:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T22:40:06.525 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T22:40:06.596 DEBUG:teuthology.orchestra.run.smithi078:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T22:40:06.650 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T22:40:06.731 DEBUG:teuthology.orchestra.run.smithi078:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T22:40:06.774 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-11T22:40:06.774 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T22:40:06.841 DEBUG:teuthology.orchestra.run.smithi132:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T22:40:06.884 DEBUG:teuthology.orchestra.run.smithi132:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T22:40:06.936 DEBUG:teuthology.orchestra.run.smithi132:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T22:40:07.017 DEBUG:teuthology.orchestra.run.smithi132:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T22:40:07.062 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:40:07.062 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T22:40:07.130 DEBUG:teuthology.orchestra.run.smithi078:> sudo service rsyslog restart 2024-03-11T22:40:07.133 DEBUG:teuthology.orchestra.run.smithi132:> sudo service rsyslog restart 2024-03-11T22:40:07.182 INFO:teuthology.orchestra.run.smithi078.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T22:40:07.219 INFO:teuthology.orchestra.run.smithi132.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T22:40:07.639 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-11T22:40:07.647 INFO:teuthology.task.internal:Starting timer... 2024-03-11T22:40:07.647 INFO:teuthology.run_tasks:Running task pcp... 2024-03-11T22:40:07.661 INFO:teuthology.run_tasks:Running task selinux... 2024-03-11T22:40:07.670 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-03-11T22:40:07.670 DEBUG:teuthology.orchestra.run.smithi078:> sudo service auditd rotate 2024-03-11T22:40:07.727 INFO:teuthology.orchestra.run.smithi078.stdout:Rotating logs: 2024-03-11T22:40:07.728 DEBUG:teuthology.orchestra.run.smithi132:> sudo service auditd rotate 2024-03-11T22:40:07.797 INFO:teuthology.orchestra.run.smithi132.stdout:Rotating logs: 2024-03-11T22:40:07.799 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-11T22:40:07.799 DEBUG:teuthology.orchestra.run.smithi078:> /usr/sbin/getenforce 2024-03-11T22:40:07.845 INFO:teuthology.orchestra.run.smithi078.stdout:Permissive 2024-03-11T22:40:07.845 DEBUG:teuthology.orchestra.run.smithi132:> /usr/sbin/getenforce 2024-03-11T22:40:07.867 INFO:teuthology.orchestra.run.smithi132.stdout:Permissive 2024-03-11T22:40:07.867 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi078.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi132.front.sepia.ceph.com': 'permissive'} 2024-03-11T22:40:07.868 DEBUG:teuthology.orchestra.run.smithi078:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-11T22:40:07.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:40:07.916 DEBUG:teuthology.orchestra.run.smithi132:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-11T22:40:07.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:40:07.943 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-11T22:40:07.943 DEBUG:teuthology.orchestra.run.smithi078:> sudo /usr/sbin/setenforce permissive 2024-03-11T22:40:07.989 DEBUG:teuthology.orchestra.run.smithi132:> sudo /usr/sbin/setenforce permissive 2024-03-11T22:40:08.019 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-11T22:40:08.029 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-11T22:40:08.035 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-11T22:40:08.158 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-11T22:40:08.173 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-11T22:40:08.174 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi078.front.sepia.ceph.com,smithi132.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-11T22:44:21.009 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi078.front.sepia.ceph.com'), Remote(name='ubuntu@smithi132.front.sepia.ceph.com')] 2024-03-11T22:44:21.011 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-03-11T22:44:21.012 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:44:21.107 DEBUG:teuthology.orchestra.run.smithi078:> true 2024-03-11T22:44:21.185 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-03-11T22:44:21.185 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-03-11T22:44:21.186 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:44:21.275 DEBUG:teuthology.orchestra.run.smithi132:> true 2024-03-11T22:44:21.352 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-03-11T22:44:21.353 INFO:teuthology.run_tasks:Running task clock... 2024-03-11T22:44:21.363 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-11T22:44:21.363 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T22:44:21.364 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T22:44:21.366 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T22:44:21.366 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T22:44:21.396 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T22:44:21.413 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T22:44:21.424 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T22:44:21.440 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T22:44:21.442 INFO:teuthology.orchestra.run.smithi078.stderr:sudo: ntpd: command not found 2024-03-11T22:44:21.456 INFO:teuthology.orchestra.run.smithi078.stdout:506 Cannot talk to daemon 2024-03-11T22:44:21.469 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: ntpd: command not found 2024-03-11T22:44:21.472 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T22:44:21.483 INFO:teuthology.orchestra.run.smithi132.stdout:506 Cannot talk to daemon 2024-03-11T22:44:21.488 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T22:44:21.500 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T22:44:21.516 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T22:44:21.542 INFO:teuthology.orchestra.run.smithi078.stderr:bash: line 1: ntpq: command not found 2024-03-11T22:44:21.545 INFO:teuthology.orchestra.run.smithi078.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T22:44:21.545 INFO:teuthology.orchestra.run.smithi078.stdout:=============================================================================== 2024-03-11T22:44:21.545 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:44:21.545 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:44:21.545 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:44:21.545 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:44:21.572 INFO:teuthology.orchestra.run.smithi132.stderr:bash: line 1: ntpq: command not found 2024-03-11T22:44:21.575 INFO:teuthology.orchestra.run.smithi132.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T22:44:21.575 INFO:teuthology.orchestra.run.smithi132.stdout:=============================================================================== 2024-03-11T22:44:21.575 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:44:21.575 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:44:21.575 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:44:21.576 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:44:21.576 INFO:teuthology.run_tasks:Running task install... 2024-03-11T22:44:21.584 DEBUG:teuthology.task.install:project ceph 2024-03-11T22:44:21.584 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'b117a47aa0ad81d207fdd9ba852068f2f262fea8'}} 2024-03-11T22:44:21.584 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'b117a47aa0ad81d207fdd9ba852068f2f262fea8'} 2024-03-11T22:44:21.585 INFO:teuthology.task.install:Using flavor: default 2024-03-11T22:44:21.611 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-11T22:44:21.611 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2024-03-11T22:44:21.612 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b117a47aa0ad81d207fdd9ba852068f2f262fea8', 'tag': None, 'wait_for_package': False} 2024-03-11T22:44:21.612 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b117a47aa0ad81d207fdd9ba852068f2f262fea8 2024-03-11T22:44:21.614 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b117a47aa0ad81d207fdd9ba852068f2f262fea8', 'tag': None, 'wait_for_package': False} 2024-03-11T22:44:21.614 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b117a47aa0ad81d207fdd9ba852068f2f262fea8 2024-03-11T22:44:21.771 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b117a47aa0ad81d207fdd9ba852068f2f262fea8/centos/9/flavors/default/ 2024-03-11T22:44:21.771 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1979.gb117a47a 2024-03-11T22:44:22.022 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b117a47aa0ad81d207fdd9ba852068f2f262fea8/centos/9/flavors/default/ 2024-03-11T22:44:22.022 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1979.gb117a47a 2024-03-11T22:44:22.072 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b117a47aa0ad81d207fdd9ba852068f2f262fea8/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b117a47aa0ad81d207fdd9ba852068f2f262fea8/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b117a47aa0ad81d207fdd9ba852068f2f262fea8/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-11T22:44:22.073 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:44:22.073 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-11T22:44:22.107 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-03-11T22:44:22.107 DEBUG:teuthology.orchestra.run.smithi132:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b117a47aa0ad81d207fdd9ba852068f2f262fea8/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-11T22:44:22.176 DEBUG:teuthology.orchestra.run.smithi132:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-11T22:44:22.254 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b117a47aa0ad81d207fdd9ba852068f2f262fea8/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b117a47aa0ad81d207fdd9ba852068f2f262fea8/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-dis-testing/b117a47aa0ad81d207fdd9ba852068f2f262fea8/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-11T22:44:22.254 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-11T22:44:22.254 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-11T22:44:22.255 DEBUG:teuthology.orchestra.run.smithi132:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-11T22:44:22.286 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-03-11T22:44:22.286 DEBUG:teuthology.orchestra.run.smithi078:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b117a47aa0ad81d207fdd9ba852068f2f262fea8/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-11T22:44:22.322 INFO:teuthology.orchestra.run.smithi132.stdout:check_obsoletes = 1 2024-03-11T22:44:22.323 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum clean all 2024-03-11T22:44:22.355 DEBUG:teuthology.orchestra.run.smithi078:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-11T22:44:22.438 DEBUG:teuthology.orchestra.run.smithi078:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-11T22:44:22.505 INFO:teuthology.orchestra.run.smithi078.stdout:check_obsoletes = 1 2024-03-11T22:44:22.507 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum clean all 2024-03-11T22:44:22.562 INFO:teuthology.orchestra.run.smithi132.stdout:74 files removed 2024-03-11T22:44:22.590 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-radosgw 2024-03-11T22:44:22.748 INFO:teuthology.orchestra.run.smithi078.stdout:74 files removed 2024-03-11T22:44:22.775 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-radosgw 2024-03-11T22:44:23.485 INFO:teuthology.orchestra.run.smithi132.stdout:ceph packages for x86_64 142 kB/s | 82 kB 00:00 2024-03-11T22:44:23.527 INFO:teuthology.orchestra.run.smithi078.stdout:ceph packages for x86_64 188 kB/s | 82 kB 00:00 2024-03-11T22:44:23.952 INFO:teuthology.orchestra.run.smithi078.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-11T22:44:24.019 INFO:teuthology.orchestra.run.smithi132.stdout:ceph noarch packages 21 kB/s | 10 kB 00:00 2024-03-11T22:44:24.362 INFO:teuthology.orchestra.run.smithi078.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-03-11T22:44:24.379 INFO:teuthology.orchestra.run.smithi132.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-03-11T22:44:24.789 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-11T22:44:24.831 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-11T22:44:26.306 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS-9 - BaseOS 5.3 MB/s | 2.0 MB 00:00 2024-03-11T22:44:26.381 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS-9 - BaseOS 5.6 MB/s | 2.0 MB 00:00 2024-03-11T22:44:27.055 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.0 MB 00:00 2024-03-11T22:44:28.772 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - AppStream 29 MB/s | 19 MB 00:00 2024-03-11T22:44:33.373 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - CRB 5.5 MB/s | 6.1 MB 00:01 2024-03-11T22:44:35.232 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 15 kB 00:00 2024-03-11T22:44:37.537 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - BaseOS 759 kB/s | 8.0 MB 00:10 2024-03-11T22:44:37.925 INFO:teuthology.orchestra.run.smithi078.stdout:Extra Packages for Enterprise Linux 8.0 MB/s | 21 MB 00:02 2024-03-11T22:44:39.696 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-03-11T22:44:42.867 INFO:teuthology.orchestra.run.smithi078.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-11T22:44:44.947 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - CRB 3.4 MB/s | 6.1 MB 00:01 2024-03-11T22:44:45.428 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-03-11T22:44:45.442 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-11T22:44:45.442 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2024-03-11T22:44:45.442 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-11T22:44:45.442 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-03-11T22:44:45.442 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-radosgw x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 9.8 M 2024-03-11T22:44:45.442 INFO:teuthology.orchestra.run.smithi078.stdout:Upgrading: 2024-03-11T22:44:45.442 INFO:teuthology.orchestra.run.smithi078.stdout: librados2 x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 3.4 M 2024-03-11T22:44:45.443 INFO:teuthology.orchestra.run.smithi078.stdout: librbd1 x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 3.1 M 2024-03-11T22:44:45.443 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-03-11T22:44:45.443 INFO:teuthology.orchestra.run.smithi078.stdout: boost-program-options 2024-03-11T22:44:45.443 INFO:teuthology.orchestra.run.smithi078.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-11T22:44:45.443 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-base x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 5.3 M 2024-03-11T22:44:45.443 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-common x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 20 M 2024-03-11T22:44:45.443 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-selinux x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 25 k 2024-03-11T22:44:45.443 INFO:teuthology.orchestra.run.smithi078.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-11T22:44:45.443 INFO:teuthology.orchestra.run.smithi078.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-11T22:44:45.443 INFO:teuthology.orchestra.run.smithi078.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-11T22:44:45.443 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs2 x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 745 k 2024-03-11T22:44:45.443 INFO:teuthology.orchestra.run.smithi078.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-11T22:44:45.443 INFO:teuthology.orchestra.run.smithi078.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-11T22:44:45.444 INFO:teuthology.orchestra.run.smithi078.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-11T22:44:45.444 INFO:teuthology.orchestra.run.smithi078.stdout: libradosstriper1 x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 501 k 2024-03-11T22:44:45.444 INFO:teuthology.orchestra.run.smithi078.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-11T22:44:45.444 INFO:teuthology.orchestra.run.smithi078.stdout: librgw2 x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 5.2 M 2024-03-11T22:44:45.444 INFO:teuthology.orchestra.run.smithi078.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-11T22:44:45.444 INFO:teuthology.orchestra.run.smithi078.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-11T22:44:45.444 INFO:teuthology.orchestra.run.smithi078.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-11T22:44:45.444 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-argparse 2024-03-11T22:44:45.444 INFO:teuthology.orchestra.run.smithi078.stdout: x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 45 k 2024-03-11T22:44:45.444 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-common x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 131 k 2024-03-11T22:44:45.444 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cephfs x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 161 k 2024-03-11T22:44:45.444 INFO:teuthology.orchestra.run.smithi078.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-11T22:44:45.444 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rados x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 321 k 2024-03-11T22:44:45.445 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rbd x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 298 k 2024-03-11T22:44:45.445 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rgw x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 100 k 2024-03-11T22:44:45.445 INFO:teuthology.orchestra.run.smithi078.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-11T22:44:45.445 INFO:teuthology.orchestra.run.smithi078.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-11T22:44:45.445 INFO:teuthology.orchestra.run.smithi078.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-11T22:44:45.445 INFO:teuthology.orchestra.run.smithi078.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-11T22:44:45.445 INFO:teuthology.orchestra.run.smithi078.stdout:Installing weak dependencies: 2024-03-11T22:44:45.445 INFO:teuthology.orchestra.run.smithi078.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-11T22:44:45.445 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:44:45.445 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-03-11T22:44:45.445 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-11T22:44:45.445 INFO:teuthology.orchestra.run.smithi078.stdout:Install 30 Packages 2024-03-11T22:44:45.445 INFO:teuthology.orchestra.run.smithi078.stdout:Upgrade 2 Packages 2024-03-11T22:44:45.445 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:44:45.446 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 58 M 2024-03-11T22:44:45.446 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-03-11T22:44:46.393 INFO:teuthology.orchestra.run.smithi078.stdout:(1/32): ceph-radosgw-19.0.0-1979.gb117a47a.el9. 13 MB/s | 9.8 MB 00:00 2024-03-11T22:44:46.435 INFO:teuthology.orchestra.run.smithi078.stdout:(2/32): ceph-selinux-19.0.0-1979.gb117a47a.el9. 605 kB/s | 25 kB 00:00 2024-03-11T22:44:46.560 INFO:teuthology.orchestra.run.smithi078.stdout:(3/32): libcephfs2-19.0.0-1979.gb117a47a.el9.x8 5.8 MB/s | 745 kB 00:00 2024-03-11T22:44:46.661 INFO:teuthology.orchestra.run.smithi078.stdout:(4/32): libradosstriper1-19.0.0-1979.gb117a47a. 4.9 MB/s | 501 kB 00:00 2024-03-11T22:44:46.714 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 15 kB 00:00 2024-03-11T22:44:46.952 INFO:teuthology.orchestra.run.smithi078.stdout:(5/32): librgw2-19.0.0-1979.gb117a47a.el9.x86_6 18 MB/s | 5.2 MB 00:00 2024-03-11T22:44:46.994 INFO:teuthology.orchestra.run.smithi078.stdout:(6/32): python3-ceph-argparse-19.0.0-1979.gb117 1.1 MB/s | 45 kB 00:00 2024-03-11T22:44:47.053 INFO:teuthology.orchestra.run.smithi078.stdout:(7/32): python3-ceph-common-19.0.0-1979.gb117a4 2.2 MB/s | 131 kB 00:00 2024-03-11T22:44:47.103 INFO:teuthology.orchestra.run.smithi078.stdout:(8/32): python3-cephfs-19.0.0-1979.gb117a47a.el 3.1 MB/s | 161 kB 00:00 2024-03-11T22:44:47.178 INFO:teuthology.orchestra.run.smithi078.stdout:(9/32): ceph-base-19.0.0-1979.gb117a47a.el9.x86 3.5 MB/s | 5.3 MB 00:01 2024-03-11T22:44:47.220 INFO:teuthology.orchestra.run.smithi078.stdout:(10/32): python3-rados-19.0.0-1979.gb117a47a.el 2.7 MB/s | 321 kB 00:00 2024-03-11T22:44:47.246 INFO:teuthology.orchestra.run.smithi078.stdout:(11/32): python3-rbd-19.0.0-1979.gb117a47a.el9. 4.4 MB/s | 298 kB 00:00 2024-03-11T22:44:47.313 INFO:teuthology.orchestra.run.smithi078.stdout:(12/32): python3-rgw-19.0.0-1979.gb117a47a.el9. 1.1 MB/s | 100 kB 00:00 2024-03-11T22:44:47.388 INFO:teuthology.orchestra.run.smithi078.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 746 kB/s | 106 kB 00:00 2024-03-11T22:44:47.413 INFO:teuthology.orchestra.run.smithi078.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.6 MB/s | 159 kB 00:00 2024-03-11T22:44:47.430 INFO:teuthology.orchestra.run.smithi078.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-11T22:44:47.497 INFO:teuthology.orchestra.run.smithi078.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2024-03-11T22:44:47.522 INFO:teuthology.orchestra.run.smithi078.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-03-11T22:44:47.539 INFO:teuthology.orchestra.run.smithi078.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 991 kB/s | 41 kB 00:00 2024-03-11T22:44:47.690 INFO:teuthology.orchestra.run.smithi078.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 232 kB/s | 35 kB 00:00 2024-03-11T22:44:47.773 INFO:teuthology.orchestra.run.smithi078.stdout:(20/32): qatlib-23.11.0-1.el9.x86_64.rpm 865 kB/s | 216 kB 00:00 2024-03-11T22:44:47.790 INFO:teuthology.orchestra.run.smithi078.stdout:(21/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 444 kB/s | 44 kB 00:00 2024-03-11T22:44:47.957 INFO:teuthology.orchestra.run.smithi078.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.6 MB/s | 309 kB 00:00 2024-03-11T22:44:47.999 INFO:teuthology.orchestra.run.smithi078.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 621 kB/s | 26 kB 00:00 2024-03-11T22:44:48.041 INFO:teuthology.orchestra.run.smithi078.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-03-11T22:44:48.125 INFO:teuthology.orchestra.run.smithi078.stdout:(25/32): libarrow-9.0.0-10.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-03-11T22:44:48.150 INFO:teuthology.orchestra.run.smithi078.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 620 kB/s | 67 kB 00:00 2024-03-11T22:44:48.184 INFO:teuthology.orchestra.run.smithi078.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 14 MB/s | 838 kB 00:00 2024-03-11T22:44:48.209 INFO:teuthology.orchestra.run.smithi078.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-03-11T22:44:48.251 INFO:teuthology.orchestra.run.smithi078.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 24 MB/s | 1.6 MB 00:00 2024-03-11T22:44:48.493 INFO:teuthology.orchestra.run.smithi078.stdout:(30/32): librbd1-19.0.0-1979.gb117a47a.el9.x86_ 13 MB/s | 3.1 MB 00:00 2024-03-11T22:44:48.541 INFO:teuthology.orchestra.run.smithi132.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-11T22:44:48.901 INFO:teuthology.orchestra.run.smithi078.stdout:(31/32): librados2-19.0.0-1979.gb117a47a.el9.x8 4.9 MB/s | 3.4 MB 00:00 2024-03-11T22:44:49.360 INFO:teuthology.orchestra.run.smithi078.stdout:(32/32): ceph-common-19.0.0-1979.gb117a47a.el9. 5.4 MB/s | 20 MB 00:03 2024-03-11T22:44:49.363 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:44:49.363 INFO:teuthology.orchestra.run.smithi078.stdout:Total 15 MB/s | 58 MB 00:03 2024-03-11T22:44:49.395 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-03-11T22:44:49.429 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-03-11T22:44:49.430 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-03-11T22:44:49.898 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-03-11T22:44:49.898 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-03-11T22:44:51.361 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-03-11T22:44:51.443 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-11T22:44:51.481 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-11T22:44:51.506 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-11T22:44:51.529 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-11T22:44:51.546 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-11T22:44:51.745 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-11T22:44:51.802 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-11T22:44:52.149 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-11T22:44:52.216 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : librados2-2:19.0.0-1979.gb117a47a.el9.x86_64 7/34 2024-03-11T22:44:52.300 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librados2-2:19.0.0-1979.gb117a47a.el9.x86_64 7/34 2024-03-11T22:44:52.397 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rados-2:19.0.0-1979.gb117a47a.el9.x86_64 8/34 2024-03-11T22:44:52.426 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephfs2-2:19.0.0-1979.gb117a47a.el9.x86_64 9/34 2024-03-11T22:44:52.489 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libcephfs2-2:19.0.0-1979.gb117a47a.el9.x86_64 9/34 2024-03-11T22:44:52.517 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-11T22:44:52.561 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-ceph-argparse-2:19.0.0-1979.gb117a47a.el9. 11/34 2024-03-11T22:44:52.629 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cephfs-2:19.0.0-1979.gb117a47a.el9.x86_64 12/34 2024-03-11T22:44:52.661 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libradosstriper1-2:19.0.0-1979.gb117a47a.el9.x86_6 13/34 2024-03-11T22:44:52.724 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1979.gb117a47a.el9.x86_6 13/34 2024-03-11T22:44:52.751 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-11T22:44:52.858 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-11T22:44:52.895 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-11T22:44:52.936 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-11T22:44:52.968 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-11T22:44:52.996 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-11T22:44:53.029 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-11T22:44:53.341 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-11T22:44:53.392 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : librbd1-2:19.0.0-1979.gb117a47a.el9.x86_64 22/34 2024-03-11T22:44:53.416 INFO:teuthology.orchestra.run.smithi132.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-11T22:44:53.482 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librbd1-2:19.0.0-1979.gb117a47a.el9.x86_64 22/34 2024-03-11T22:44:53.513 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rbd-2:19.0.0-1979.gb117a47a.el9.x86_64 23/34 2024-03-11T22:44:53.558 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-11T22:44:54.015 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-11T22:44:54.045 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librgw2-2:19.0.0-1979.gb117a47a.el9.x86_64 26/34 2024-03-11T22:44:54.099 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librgw2-2:19.0.0-1979.gb117a47a.el9.x86_64 26/34 2024-03-11T22:44:54.166 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rgw-2:19.0.0-1979.gb117a47a.el9.x86_64 27/34 2024-03-11T22:44:54.197 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-ceph-common-2:19.0.0-1979.gb117a47a.el9.x8 28/34 2024-03-11T22:44:55.952 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-11T22:44:55.955 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-11T22:44:55.955 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-03-11T22:44:55.955 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-11T22:44:55.955 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-11T22:44:55.955 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-radosgw x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 9.8 M 2024-03-11T22:44:55.956 INFO:teuthology.orchestra.run.smithi132.stdout:Upgrading: 2024-03-11T22:44:55.956 INFO:teuthology.orchestra.run.smithi132.stdout: librados2 x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 3.4 M 2024-03-11T22:44:55.956 INFO:teuthology.orchestra.run.smithi132.stdout: librbd1 x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 3.1 M 2024-03-11T22:44:55.956 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-03-11T22:44:55.956 INFO:teuthology.orchestra.run.smithi132.stdout: boost-program-options 2024-03-11T22:44:55.956 INFO:teuthology.orchestra.run.smithi132.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-11T22:44:55.956 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-base x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 5.3 M 2024-03-11T22:44:55.956 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-common x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 20 M 2024-03-11T22:44:55.956 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-selinux x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 25 k 2024-03-11T22:44:55.956 INFO:teuthology.orchestra.run.smithi132.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-11T22:44:55.956 INFO:teuthology.orchestra.run.smithi132.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-11T22:44:55.956 INFO:teuthology.orchestra.run.smithi132.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-11T22:44:55.956 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs2 x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 745 k 2024-03-11T22:44:55.956 INFO:teuthology.orchestra.run.smithi132.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-11T22:44:55.957 INFO:teuthology.orchestra.run.smithi132.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-11T22:44:55.957 INFO:teuthology.orchestra.run.smithi132.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-11T22:44:55.957 INFO:teuthology.orchestra.run.smithi132.stdout: libradosstriper1 x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 501 k 2024-03-11T22:44:55.957 INFO:teuthology.orchestra.run.smithi132.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-11T22:44:55.957 INFO:teuthology.orchestra.run.smithi132.stdout: librgw2 x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 5.2 M 2024-03-11T22:44:55.957 INFO:teuthology.orchestra.run.smithi132.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-11T22:44:55.957 INFO:teuthology.orchestra.run.smithi132.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-11T22:44:55.957 INFO:teuthology.orchestra.run.smithi132.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-11T22:44:55.957 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-argparse 2024-03-11T22:44:55.957 INFO:teuthology.orchestra.run.smithi132.stdout: x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 45 k 2024-03-11T22:44:55.957 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-common x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 131 k 2024-03-11T22:44:55.957 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cephfs x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 161 k 2024-03-11T22:44:55.957 INFO:teuthology.orchestra.run.smithi132.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-11T22:44:55.957 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rados x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 321 k 2024-03-11T22:44:55.958 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rbd x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 298 k 2024-03-11T22:44:55.958 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rgw x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 100 k 2024-03-11T22:44:55.958 INFO:teuthology.orchestra.run.smithi132.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-11T22:44:55.958 INFO:teuthology.orchestra.run.smithi132.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-11T22:44:55.958 INFO:teuthology.orchestra.run.smithi132.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-11T22:44:55.958 INFO:teuthology.orchestra.run.smithi132.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-11T22:44:55.958 INFO:teuthology.orchestra.run.smithi132.stdout:Installing weak dependencies: 2024-03-11T22:44:55.958 INFO:teuthology.orchestra.run.smithi132.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-11T22:44:55.958 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:44:55.958 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-11T22:44:55.958 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-11T22:44:55.958 INFO:teuthology.orchestra.run.smithi132.stdout:Install 30 Packages 2024-03-11T22:44:55.958 INFO:teuthology.orchestra.run.smithi132.stdout:Upgrade 2 Packages 2024-03-11T22:44:55.958 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:44:55.959 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 58 M 2024-03-11T22:44:55.959 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-11T22:44:56.373 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-common-2:19.0.0-1979.gb117a47a.el9.x86_64 29/34 2024-03-11T22:44:56.418 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-common-2:19.0.0-1979.gb117a47a.el9.x86_64 29/34 2024-03-11T22:44:56.992 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-common-2:19.0.0-1979.gb117a47a.el9.x86_64 29/34 2024-03-11T22:44:57.030 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-base-2:19.0.0-1979.gb117a47a.el9.x86_64 30/34 2024-03-11T22:44:57.127 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-base-2:19.0.0-1979.gb117a47a.el9.x86_64 30/34 2024-03-11T22:44:57.127 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-11T22:44:57.127 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-11T22:44:57.127 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:44:57.164 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-selinux-2:19.0.0-1979.gb117a47a.el9.x86_64 31/34 2024-03-11T22:44:58.100 INFO:teuthology.orchestra.run.smithi132.stdout:(1/32): ceph-base-19.0.0-1979.gb117a47a.el9.x86 2.8 MB/s | 5.3 MB 00:01 2024-03-11T22:44:58.159 INFO:teuthology.orchestra.run.smithi132.stdout:(2/32): ceph-selinux-19.0.0-1979.gb117a47a.el9. 433 kB/s | 25 kB 00:00 2024-03-11T22:44:58.301 INFO:teuthology.orchestra.run.smithi132.stdout:(3/32): libcephfs2-19.0.0-1979.gb117a47a.el9.x8 5.1 MB/s | 745 kB 00:00 2024-03-11T22:44:58.417 INFO:teuthology.orchestra.run.smithi132.stdout:(4/32): libradosstriper1-19.0.0-1979.gb117a47a. 4.2 MB/s | 501 kB 00:00 2024-03-11T22:44:58.868 INFO:teuthology.orchestra.run.smithi132.stdout:(5/32): ceph-common-19.0.0-1979.gb117a47a.el9.x 7.5 MB/s | 20 MB 00:02 2024-03-11T22:44:58.918 INFO:teuthology.orchestra.run.smithi132.stdout:(6/32): python3-ceph-argparse-19.0.0-1979.gb117 908 kB/s | 45 kB 00:00 2024-03-11T22:44:58.985 INFO:teuthology.orchestra.run.smithi132.stdout:(7/32): python3-ceph-common-19.0.0-1979.gb117a4 1.9 MB/s | 131 kB 00:00 2024-03-11T22:44:59.052 INFO:teuthology.orchestra.run.smithi132.stdout:(8/32): librgw2-19.0.0-1979.gb117a47a.el9.x86_6 8.1 MB/s | 5.2 MB 00:00 2024-03-11T22:44:59.077 INFO:teuthology.orchestra.run.smithi132.stdout:(9/32): python3-cephfs-19.0.0-1979.gb117a47a.el 1.7 MB/s | 161 kB 00:00 2024-03-11T22:44:59.102 INFO:teuthology.orchestra.run.smithi132.stdout:(10/32): python3-rados-19.0.0-1979.gb117a47a.el 6.3 MB/s | 321 kB 00:00 2024-03-11T22:44:59.136 INFO:teuthology.orchestra.run.smithi132.stdout:(11/32): python3-rbd-19.0.0-1979.gb117a47a.el9. 5.0 MB/s | 298 kB 00:00 2024-03-11T22:44:59.161 INFO:teuthology.orchestra.run.smithi132.stdout:(12/32): python3-rgw-19.0.0-1979.gb117a47a.el9. 1.7 MB/s | 100 kB 00:00 2024-03-11T22:44:59.312 INFO:teuthology.orchestra.run.smithi132.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 604 kB/s | 106 kB 00:00 2024-03-11T22:44:59.337 INFO:teuthology.orchestra.run.smithi132.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-03-11T22:44:59.362 INFO:teuthology.orchestra.run.smithi132.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 888 kB/s | 44 kB 00:00 2024-03-11T22:44:59.421 INFO:teuthology.orchestra.run.smithi132.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2024-03-11T22:44:59.454 INFO:teuthology.orchestra.run.smithi132.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-03-11T22:44:59.480 INFO:teuthology.orchestra.run.smithi132.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 705 kB/s | 41 kB 00:00 2024-03-11T22:44:59.597 INFO:teuthology.orchestra.run.smithi132.stdout:(19/32): ceph-radosgw-19.0.0-1979.gb117a47a.el9 2.9 MB/s | 9.8 MB 00:03 2024-03-11T22:44:59.664 INFO:teuthology.orchestra.run.smithi132.stdout:(20/32): qatlib-service-23.11.0-1.el9.x86_64.rp 190 kB/s | 35 kB 00:00 2024-03-11T22:44:59.706 INFO:teuthology.orchestra.run.smithi132.stdout:(21/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 411 kB/s | 44 kB 00:00 2024-03-11T22:44:59.731 INFO:teuthology.orchestra.run.smithi132.stdout:(22/32): qatlib-23.11.0-1.el9.x86_64.rpm 785 kB/s | 216 kB 00:00 2024-03-11T22:44:59.856 INFO:teuthology.orchestra.run.smithi132.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 208 kB/s | 26 kB 00:00 2024-03-11T22:44:59.948 INFO:teuthology.orchestra.run.smithi132.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.1 MB/s | 309 kB 00:00 2024-03-11T22:44:59.982 INFO:teuthology.orchestra.run.smithi132.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 388 kB/s | 49 kB 00:00 2024-03-11T22:45:00.007 INFO:teuthology.orchestra.run.smithi132.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-11T22:45:00.074 INFO:teuthology.orchestra.run.smithi132.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-11T22:45:00.158 INFO:teuthology.orchestra.run.smithi132.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 9.8 MB/s | 4.4 MB 00:00 2024-03-11T22:45:00.249 INFO:teuthology.orchestra.run.smithi132.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.1 MB/s | 838 kB 00:00 2024-03-11T22:45:00.283 INFO:teuthology.orchestra.run.smithi132.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 7.6 MB/s | 1.6 MB 00:00 2024-03-11T22:45:00.508 INFO:teuthology.orchestra.run.smithi132.stdout:(31/32): librados2-19.0.0-1979.gb117a47a.el9.x8 9.7 MB/s | 3.4 MB 00:00 2024-03-11T22:45:00.567 INFO:teuthology.orchestra.run.smithi132.stdout:(32/32): librbd1-19.0.0-1979.gb117a47a.el9.x86_ 9.9 MB/s | 3.1 MB 00:00 2024-03-11T22:45:00.571 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:45:00.571 INFO:teuthology.orchestra.run.smithi132.stdout:Total 13 MB/s | 58 MB 00:04 2024-03-11T22:45:00.607 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-11T22:45:00.644 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-11T22:45:00.644 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-11T22:45:01.155 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-11T22:45:01.155 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-11T22:45:02.582 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-11T22:45:02.657 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-11T22:45:02.696 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-11T22:45:02.722 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-11T22:45:02.744 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-11T22:45:02.761 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-11T22:45:02.927 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-11T22:45:02.983 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-11T22:45:03.409 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-11T22:45:03.440 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : librados2-2:19.0.0-1979.gb117a47a.el9.x86_64 7/34 2024-03-11T22:45:03.523 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados2-2:19.0.0-1979.gb117a47a.el9.x86_64 7/34 2024-03-11T22:45:03.614 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rados-2:19.0.0-1979.gb117a47a.el9.x86_64 8/34 2024-03-11T22:45:03.650 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephfs2-2:19.0.0-1979.gb117a47a.el9.x86_64 9/34 2024-03-11T22:45:03.713 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libcephfs2-2:19.0.0-1979.gb117a47a.el9.x86_64 9/34 2024-03-11T22:45:03.750 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-11T22:45:03.798 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-ceph-argparse-2:19.0.0-1979.gb117a47a.el9. 11/34 2024-03-11T22:45:03.872 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cephfs-2:19.0.0-1979.gb117a47a.el9.x86_64 12/34 2024-03-11T22:45:03.894 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libradosstriper1-2:19.0.0-1979.gb117a47a.el9.x86_6 13/34 2024-03-11T22:45:03.941 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1979.gb117a47a.el9.x86_6 13/34 2024-03-11T22:45:03.968 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-11T22:45:04.078 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-11T22:45:04.144 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-11T22:45:04.184 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-11T22:45:04.217 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-11T22:45:04.246 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-11T22:45:04.279 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-11T22:45:04.602 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-11T22:45:04.625 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : librbd1-2:19.0.0-1979.gb117a47a.el9.x86_64 22/34 2024-03-11T22:45:04.716 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librbd1-2:19.0.0-1979.gb117a47a.el9.x86_64 22/34 2024-03-11T22:45:04.747 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rbd-2:19.0.0-1979.gb117a47a.el9.x86_64 23/34 2024-03-11T22:45:04.786 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-11T22:45:05.246 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-11T22:45:05.287 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librgw2-2:19.0.0-1979.gb117a47a.el9.x86_64 26/34 2024-03-11T22:45:05.340 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librgw2-2:19.0.0-1979.gb117a47a.el9.x86_64 26/34 2024-03-11T22:45:05.409 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rgw-2:19.0.0-1979.gb117a47a.el9.x86_64 27/34 2024-03-11T22:45:05.439 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-ceph-common-2:19.0.0-1979.gb117a47a.el9.x8 28/34 2024-03-11T22:45:06.687 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1979.gb117a47a.el9.x86_64 31/34 2024-03-11T22:45:06.688 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /sys 2024-03-11T22:45:06.688 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /proc 2024-03-11T22:45:06.688 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /mnt 2024-03-11T22:45:06.688 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /var/tmp 2024-03-11T22:45:06.688 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /home 2024-03-11T22:45:06.688 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /root 2024-03-11T22:45:06.688 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /tmp 2024-03-11T22:45:06.688 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:45:07.614 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-common-2:19.0.0-1979.gb117a47a.el9.x86_64 29/34 2024-03-11T22:45:07.614 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-radosgw-2:19.0.0-1979.gb117a47a.el9.x86_64 32/34 2024-03-11T22:45:07.638 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1979.gb117a47a.el9.x86_64 32/34 2024-03-11T22:45:07.638 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:45:07.638 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-11T22:45:07.638 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T22:45:07.639 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T22:45:07.639 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:45:07.640 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-11T22:45:07.660 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-common-2:19.0.0-1979.gb117a47a.el9.x86_64 29/34 2024-03-11T22:45:07.725 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-11T22:45:07.725 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-11T22:45:08.223 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-common-2:19.0.0-1979.gb117a47a.el9.x86_64 29/34 2024-03-11T22:45:08.272 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-base-2:19.0.0-1979.gb117a47a.el9.x86_64 30/34 2024-03-11T22:45:08.348 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-base-2:19.0.0-1979.gb117a47a.el9.x86_64 30/34 2024-03-11T22:45:08.348 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-11T22:45:08.348 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-11T22:45:08.348 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:45:08.390 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-selinux-2:19.0.0-1979.gb117a47a.el9.x86_64 31/34 2024-03-11T22:45:08.832 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-11T22:45:08.833 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-base-2:19.0.0-1979.gb117a47a.el9.x86_64 1/34 2024-03-11T22:45:08.833 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-common-2:19.0.0-1979.gb117a47a.el9.x86_64 2/34 2024-03-11T22:45:08.833 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-radosgw-2:19.0.0-1979.gb117a47a.el9.x86_64 3/34 2024-03-11T22:45:08.833 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-selinux-2:19.0.0-1979.gb117a47a.el9.x86_64 4/34 2024-03-11T22:45:08.833 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephfs2-2:19.0.0-1979.gb117a47a.el9.x86_64 5/34 2024-03-11T22:45:08.833 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libradosstriper1-2:19.0.0-1979.gb117a47a.el9.x86_6 6/34 2024-03-11T22:45:08.833 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librgw2-2:19.0.0-1979.gb117a47a.el9.x86_64 7/34 2024-03-11T22:45:08.833 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1979.gb117a47a.el9. 8/34 2024-03-11T22:45:08.833 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-ceph-common-2:19.0.0-1979.gb117a47a.el9.x8 9/34 2024-03-11T22:45:08.833 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cephfs-2:19.0.0-1979.gb117a47a.el9.x86_64 10/34 2024-03-11T22:45:08.834 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rados-2:19.0.0-1979.gb117a47a.el9.x86_64 11/34 2024-03-11T22:45:08.834 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rbd-2:19.0.0-1979.gb117a47a.el9.x86_64 12/34 2024-03-11T22:45:08.834 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rgw-2:19.0.0-1979.gb117a47a.el9.x86_64 13/34 2024-03-11T22:45:08.834 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-11T22:45:08.834 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-11T22:45:08.834 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-11T22:45:08.834 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-11T22:45:08.834 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-11T22:45:08.834 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-11T22:45:08.834 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-11T22:45:08.835 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-11T22:45:08.835 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-11T22:45:08.835 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-11T22:45:08.835 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-11T22:45:08.835 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-11T22:45:08.835 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-11T22:45:08.835 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-11T22:45:08.835 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-11T22:45:08.835 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-11T22:45:08.835 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-11T22:45:08.835 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados2-2:19.0.0-1979.gb117a47a.el9.x86_64 31/34 2024-03-11T22:45:08.835 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-11T22:45:08.835 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librbd1-2:19.0.0-1979.gb117a47a.el9.x86_64 33/34 2024-03-11T22:45:10.098 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-11T22:45:10.098 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:45:10.098 INFO:teuthology.orchestra.run.smithi078.stdout:Upgraded: 2024-03-11T22:45:10.098 INFO:teuthology.orchestra.run.smithi078.stdout: librados2-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:10.098 INFO:teuthology.orchestra.run.smithi078.stdout: librbd1-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:10.098 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-03-11T22:45:10.098 INFO:teuthology.orchestra.run.smithi078.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-11T22:45:10.098 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-base-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:10.098 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-common-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:10.098 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-radosgw-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:10.099 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-selinux-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:10.099 INFO:teuthology.orchestra.run.smithi078.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-11T22:45:10.099 INFO:teuthology.orchestra.run.smithi078.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-11T22:45:10.099 INFO:teuthology.orchestra.run.smithi078.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-11T22:45:10.099 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs2-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:10.099 INFO:teuthology.orchestra.run.smithi078.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-11T22:45:10.099 INFO:teuthology.orchestra.run.smithi078.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-11T22:45:10.099 INFO:teuthology.orchestra.run.smithi078.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-11T22:45:10.099 INFO:teuthology.orchestra.run.smithi078.stdout: libradosstriper1-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:10.099 INFO:teuthology.orchestra.run.smithi078.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-11T22:45:10.099 INFO:teuthology.orchestra.run.smithi078.stdout: librgw2-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:10.099 INFO:teuthology.orchestra.run.smithi078.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-11T22:45:10.099 INFO:teuthology.orchestra.run.smithi078.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-11T22:45:10.099 INFO:teuthology.orchestra.run.smithi078.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-11T22:45:10.100 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-argparse-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:10.100 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-common-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:10.100 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cephfs-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:10.100 INFO:teuthology.orchestra.run.smithi078.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-11T22:45:10.100 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rados-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:10.100 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rbd-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:10.100 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rgw-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:10.100 INFO:teuthology.orchestra.run.smithi078.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-11T22:45:10.100 INFO:teuthology.orchestra.run.smithi078.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-11T22:45:10.100 INFO:teuthology.orchestra.run.smithi078.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-11T22:45:10.100 INFO:teuthology.orchestra.run.smithi078.stdout: re2-1:20211101-3.el9.x86_64 2024-03-11T22:45:10.100 INFO:teuthology.orchestra.run.smithi078.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-11T22:45:10.100 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:45:10.100 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-03-11T22:45:10.480 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-test 2024-03-11T22:45:11.041 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:00:29 ago on Mon 11 Mar 2024 10:44:42 PM UTC. 2024-03-11T22:45:11.192 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-03-11T22:45:11.193 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-11T22:45:11.193 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2024-03-11T22:45:11.193 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-11T22:45:11.193 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-03-11T22:45:11.193 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-test x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 48 M 2024-03-11T22:45:11.193 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-03-11T22:45:11.193 INFO:teuthology.orchestra.run.smithi078.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-11T22:45:11.193 INFO:teuthology.orchestra.run.smithi078.stdout: libcephsqlite x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 163 k 2024-03-11T22:45:11.193 INFO:teuthology.orchestra.run.smithi078.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-11T22:45:11.194 INFO:teuthology.orchestra.run.smithi078.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-11T22:45:11.194 INFO:teuthology.orchestra.run.smithi078.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-11T22:45:11.194 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:45:11.194 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-03-11T22:45:11.194 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-11T22:45:11.194 INFO:teuthology.orchestra.run.smithi078.stdout:Install 6 Packages 2024-03-11T22:45:11.194 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:45:11.195 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 48 M 2024-03-11T22:45:11.198 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 205 M 2024-03-11T22:45:11.198 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-03-11T22:45:11.527 INFO:teuthology.orchestra.run.smithi078.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.0 MB/s | 217 kB 00:00 2024-03-11T22:45:11.652 INFO:teuthology.orchestra.run.smithi078.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.3 MB/s | 300 kB 00:00 2024-03-11T22:45:11.711 INFO:teuthology.orchestra.run.smithi078.stdout:(3/6): libcephsqlite-19.0.0-1979.gb117a47a.el9. 413 kB/s | 163 kB 00:00 2024-03-11T22:45:11.737 INFO:teuthology.orchestra.run.smithi078.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 746 kB/s | 63 kB 00:00 2024-03-11T22:45:11.912 INFO:teuthology.orchestra.run.smithi078.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 935 kB/s | 187 kB 00:00 2024-03-11T22:45:15.404 INFO:teuthology.orchestra.run.smithi078.stdout:(6/6): ceph-test-19.0.0-1979.gb117a47a.el9.x86_ 12 MB/s | 48 MB 00:04 2024-03-11T22:45:15.407 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:45:15.407 INFO:teuthology.orchestra.run.smithi078.stdout:Total 12 MB/s | 48 MB 00:04 2024-03-11T22:45:15.421 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-03-11T22:45:15.444 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-03-11T22:45:15.444 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-03-11T22:45:15.833 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-03-11T22:45:15.834 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-03-11T22:45:16.406 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-03-11T22:45:16.475 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-11T22:45:16.512 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-11T22:45:16.545 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-11T22:45:16.586 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-11T22:45:16.615 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephsqlite-2:19.0.0-1979.gb117a47a.el9.x86_64 5/6 2024-03-11T22:45:18.103 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1979.gb117a47a.el9.x86_64 31/34 2024-03-11T22:45:18.103 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /sys 2024-03-11T22:45:18.103 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /proc 2024-03-11T22:45:18.103 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /mnt 2024-03-11T22:45:18.103 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /var/tmp 2024-03-11T22:45:18.103 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /home 2024-03-11T22:45:18.103 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /root 2024-03-11T22:45:18.103 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /tmp 2024-03-11T22:45:18.103 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:45:19.028 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-radosgw-2:19.0.0-1979.gb117a47a.el9.x86_64 32/34 2024-03-11T22:45:19.059 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1979.gb117a47a.el9.x86_64 32/34 2024-03-11T22:45:19.059 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:45:19.059 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-11T22:45:19.059 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T22:45:19.060 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T22:45:19.060 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:45:19.061 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-11T22:45:19.131 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-11T22:45:19.131 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-11T22:45:20.250 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-11T22:45:20.250 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-base-2:19.0.0-1979.gb117a47a.el9.x86_64 1/34 2024-03-11T22:45:20.250 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-common-2:19.0.0-1979.gb117a47a.el9.x86_64 2/34 2024-03-11T22:45:20.250 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-radosgw-2:19.0.0-1979.gb117a47a.el9.x86_64 3/34 2024-03-11T22:45:20.250 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-selinux-2:19.0.0-1979.gb117a47a.el9.x86_64 4/34 2024-03-11T22:45:20.250 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephfs2-2:19.0.0-1979.gb117a47a.el9.x86_64 5/34 2024-03-11T22:45:20.250 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libradosstriper1-2:19.0.0-1979.gb117a47a.el9.x86_6 6/34 2024-03-11T22:45:20.250 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librgw2-2:19.0.0-1979.gb117a47a.el9.x86_64 7/34 2024-03-11T22:45:20.251 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1979.gb117a47a.el9. 8/34 2024-03-11T22:45:20.251 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-ceph-common-2:19.0.0-1979.gb117a47a.el9.x8 9/34 2024-03-11T22:45:20.251 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cephfs-2:19.0.0-1979.gb117a47a.el9.x86_64 10/34 2024-03-11T22:45:20.251 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rados-2:19.0.0-1979.gb117a47a.el9.x86_64 11/34 2024-03-11T22:45:20.251 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rbd-2:19.0.0-1979.gb117a47a.el9.x86_64 12/34 2024-03-11T22:45:20.251 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rgw-2:19.0.0-1979.gb117a47a.el9.x86_64 13/34 2024-03-11T22:45:20.251 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-11T22:45:20.251 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-11T22:45:20.251 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-11T22:45:20.251 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-11T22:45:20.251 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-11T22:45:20.251 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-11T22:45:20.251 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-11T22:45:20.251 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-11T22:45:20.252 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-11T22:45:20.252 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-11T22:45:20.252 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-11T22:45:20.252 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-11T22:45:20.252 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-11T22:45:20.252 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-11T22:45:20.252 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-11T22:45:20.252 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-11T22:45:20.252 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-11T22:45:20.254 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados2-2:19.0.0-1979.gb117a47a.el9.x86_64 31/34 2024-03-11T22:45:20.254 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-11T22:45:20.254 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librbd1-2:19.0.0-1979.gb117a47a.el9.x86_64 33/34 2024-03-11T22:45:21.125 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1979.gb117a47a.el9.x86_64 5/6 2024-03-11T22:45:21.207 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-test-2:19.0.0-1979.gb117a47a.el9.x86_64 6/6 2024-03-11T22:45:21.380 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-11T22:45:21.380 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:45:21.380 INFO:teuthology.orchestra.run.smithi132.stdout:Upgraded: 2024-03-11T22:45:21.380 INFO:teuthology.orchestra.run.smithi132.stdout: librados2-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:21.380 INFO:teuthology.orchestra.run.smithi132.stdout: librbd1-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:21.380 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-11T22:45:21.380 INFO:teuthology.orchestra.run.smithi132.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-11T22:45:21.380 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-base-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:21.380 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-common-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:21.381 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-radosgw-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:21.381 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-selinux-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:21.381 INFO:teuthology.orchestra.run.smithi132.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-11T22:45:21.381 INFO:teuthology.orchestra.run.smithi132.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-11T22:45:21.381 INFO:teuthology.orchestra.run.smithi132.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-11T22:45:21.381 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs2-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:21.381 INFO:teuthology.orchestra.run.smithi132.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-11T22:45:21.381 INFO:teuthology.orchestra.run.smithi132.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-11T22:45:21.381 INFO:teuthology.orchestra.run.smithi132.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-11T22:45:21.381 INFO:teuthology.orchestra.run.smithi132.stdout: libradosstriper1-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:21.381 INFO:teuthology.orchestra.run.smithi132.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-11T22:45:21.381 INFO:teuthology.orchestra.run.smithi132.stdout: librgw2-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:21.381 INFO:teuthology.orchestra.run.smithi132.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-11T22:45:21.381 INFO:teuthology.orchestra.run.smithi132.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-11T22:45:21.382 INFO:teuthology.orchestra.run.smithi132.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-11T22:45:21.382 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-argparse-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:21.382 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-common-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:21.382 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cephfs-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:21.382 INFO:teuthology.orchestra.run.smithi132.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-11T22:45:21.382 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rados-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:21.382 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rbd-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:21.382 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rgw-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:21.382 INFO:teuthology.orchestra.run.smithi132.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-11T22:45:21.382 INFO:teuthology.orchestra.run.smithi132.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-11T22:45:21.382 INFO:teuthology.orchestra.run.smithi132.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-11T22:45:21.382 INFO:teuthology.orchestra.run.smithi132.stdout: re2-1:20211101-3.el9.x86_64 2024-03-11T22:45:21.382 INFO:teuthology.orchestra.run.smithi132.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-11T22:45:21.382 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:45:21.383 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-11T22:45:21.648 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-test 2024-03-11T22:45:22.211 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:00:29 ago on Mon 11 Mar 2024 10:44:53 PM UTC. 2024-03-11T22:45:22.361 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-11T22:45:22.362 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-11T22:45:22.362 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-03-11T22:45:22.362 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-11T22:45:22.362 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-11T22:45:22.362 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-test x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 48 M 2024-03-11T22:45:22.362 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-03-11T22:45:22.362 INFO:teuthology.orchestra.run.smithi132.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-11T22:45:22.362 INFO:teuthology.orchestra.run.smithi132.stdout: libcephsqlite x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 163 k 2024-03-11T22:45:22.362 INFO:teuthology.orchestra.run.smithi132.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-11T22:45:22.363 INFO:teuthology.orchestra.run.smithi132.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-11T22:45:22.363 INFO:teuthology.orchestra.run.smithi132.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-11T22:45:22.363 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:45:22.363 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-11T22:45:22.363 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-11T22:45:22.363 INFO:teuthology.orchestra.run.smithi132.stdout:Install 6 Packages 2024-03-11T22:45:22.363 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:45:22.363 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 48 M 2024-03-11T22:45:22.364 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 205 M 2024-03-11T22:45:22.364 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-11T22:45:22.606 INFO:teuthology.orchestra.run.smithi132.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-11T22:45:22.665 INFO:teuthology.orchestra.run.smithi132.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-11T22:45:22.707 INFO:teuthology.orchestra.run.smithi132.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-11T22:45:22.824 INFO:teuthology.orchestra.run.smithi132.stdout:(4/6): libcephsqlite-19.0.0-1979.gb117a47a.el9. 410 kB/s | 163 kB 00:00 2024-03-11T22:45:22.849 INFO:teuthology.orchestra.run.smithi132.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-11T22:45:23.649 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-test-2:19.0.0-1979.gb117a47a.el9.x86_64 6/6 2024-03-11T22:45:23.649 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-test-2:19.0.0-1979.gb117a47a.el9.x86_64 1/6 2024-03-11T22:45:23.649 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephsqlite-2:19.0.0-1979.gb117a47a.el9.x86_64 2/6 2024-03-11T22:45:23.649 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-11T22:45:23.649 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-11T22:45:23.649 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-11T22:45:24.039 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-11T22:45:24.039 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:45:24.039 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-03-11T22:45:24.040 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-test-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:24.040 INFO:teuthology.orchestra.run.smithi078.stdout: jq-1.6-16.el9.x86_64 2024-03-11T22:45:24.040 INFO:teuthology.orchestra.run.smithi078.stdout: libcephsqlite-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:24.041 INFO:teuthology.orchestra.run.smithi078.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-11T22:45:24.041 INFO:teuthology.orchestra.run.smithi078.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-11T22:45:24.041 INFO:teuthology.orchestra.run.smithi078.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-11T22:45:24.041 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:45:24.041 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-03-11T22:45:24.348 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph 2024-03-11T22:45:24.906 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:00:42 ago on Mon 11 Mar 2024 10:44:42 PM UTC. 2024-03-11T22:45:25.060 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-03-11T22:45:25.062 INFO:teuthology.orchestra.run.smithi078.stdout:========================================================================================= 2024-03-11T22:45:25.062 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2024-03-11T22:45:25.063 INFO:teuthology.orchestra.run.smithi078.stdout:========================================================================================= 2024-03-11T22:45:25.063 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-03-11T22:45:25.063 INFO:teuthology.orchestra.run.smithi078.stdout: ceph x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 6.6 k 2024-03-11T22:45:25.063 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-03-11T22:45:25.063 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mds x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 2.1 M 2024-03-11T22:45:25.063 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 1.5 M 2024-03-11T22:45:25.063 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1979.gb117a47a.el9 ceph-noarch 248 k 2024-03-11T22:45:25.063 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mon x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 4.7 M 2024-03-11T22:45:25.063 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-osd x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 17 M 2024-03-11T22:45:25.063 INFO:teuthology.orchestra.run.smithi078.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-11T22:45:25.063 INFO:teuthology.orchestra.run.smithi078.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-11T22:45:25.063 INFO:teuthology.orchestra.run.smithi078.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-11T22:45:25.063 INFO:teuthology.orchestra.run.smithi078.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-11T22:45:25.064 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-11T22:45:25.064 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-11T22:45:25.064 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-11T22:45:25.064 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-11T22:45:25.064 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-11T22:45:25.064 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-11T22:45:25.064 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-11T22:45:25.064 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-11T22:45:25.064 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-11T22:45:25.064 INFO:teuthology.orchestra.run.smithi078.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-11T22:45:25.064 INFO:teuthology.orchestra.run.smithi078.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-11T22:45:25.064 INFO:teuthology.orchestra.run.smithi078.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-11T22:45:25.064 INFO:teuthology.orchestra.run.smithi078.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-11T22:45:25.064 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-11T22:45:25.065 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-11T22:45:25.065 INFO:teuthology.orchestra.run.smithi078.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-11T22:45:25.065 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-11T22:45:25.065 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-11T22:45:25.065 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-11T22:45:25.065 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-11T22:45:25.065 INFO:teuthology.orchestra.run.smithi078.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-11T22:45:25.065 INFO:teuthology.orchestra.run.smithi078.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-11T22:45:25.065 INFO:teuthology.orchestra.run.smithi078.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-11T22:45:25.065 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-11T22:45:25.065 INFO:teuthology.orchestra.run.smithi078.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-11T22:45:25.065 INFO:teuthology.orchestra.run.smithi078.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-11T22:45:25.065 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:45:25.065 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-03-11T22:45:25.066 INFO:teuthology.orchestra.run.smithi078.stdout:========================================================================================= 2024-03-11T22:45:25.066 INFO:teuthology.orchestra.run.smithi078.stdout:Install 36 Packages 2024-03-11T22:45:25.066 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:45:25.067 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 30 M 2024-03-11T22:45:25.067 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 107 M 2024-03-11T22:45:25.067 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-03-11T22:45:25.307 INFO:teuthology.orchestra.run.smithi132.stdout:(6/6): ceph-test-19.0.0-1979.gb117a47a.el9.x86_ 17 MB/s | 48 MB 00:02 2024-03-11T22:45:25.310 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:45:25.311 INFO:teuthology.orchestra.run.smithi132.stdout:Total 16 MB/s | 48 MB 00:02 2024-03-11T22:45:25.326 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-11T22:45:25.347 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-11T22:45:25.347 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-11T22:45:25.739 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-11T22:45:25.739 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-11T22:45:25.925 INFO:teuthology.orchestra.run.smithi078.stdout:(1/36): ceph-19.0.0-1979.gb117a47a.el9.x86_64.r 23 kB/s | 6.6 kB 00:00 2024-03-11T22:45:26.025 INFO:teuthology.orchestra.run.smithi078.stdout:(2/36): ceph-mds-19.0.0-1979.gb117a47a.el9.x86_ 5.6 MB/s | 2.1 MB 00:00 2024-03-11T22:45:26.067 INFO:teuthology.orchestra.run.smithi078.stdout:(3/36): ceph-mgr-19.0.0-1979.gb117a47a.el9.x86_ 3.5 MB/s | 1.5 MB 00:00 2024-03-11T22:45:26.242 INFO:teuthology.orchestra.run.smithi078.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1979.gb117 1.4 MB/s | 248 kB 00:00 2024-03-11T22:45:26.306 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-11T22:45:26.379 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-11T22:45:26.417 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-11T22:45:26.417 INFO:teuthology.orchestra.run.smithi078.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-11T22:45:26.449 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-11T22:45:26.468 INFO:teuthology.orchestra.run.smithi078.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-11T22:45:26.490 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-11T22:45:26.519 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephsqlite-2:19.0.0-1979.gb117a47a.el9.x86_64 5/6 2024-03-11T22:45:26.526 INFO:teuthology.orchestra.run.smithi078.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 548 kB/s | 32 kB 00:00 2024-03-11T22:45:26.568 INFO:teuthology.orchestra.run.smithi078.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-11T22:45:26.610 INFO:teuthology.orchestra.run.smithi078.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 928 kB/s | 39 kB 00:00 2024-03-11T22:45:26.660 INFO:teuthology.orchestra.run.smithi078.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-11T22:45:26.836 INFO:teuthology.orchestra.run.smithi078.stdout:(11/36): ceph-osd-19.0.0-1979.gb117a47a.el9.x86 21 MB/s | 17 MB 00:00 2024-03-11T22:45:26.869 INFO:teuthology.orchestra.run.smithi078.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 5.5 MB/s | 1.1 MB 00:00 2024-03-11T22:45:26.895 INFO:teuthology.orchestra.run.smithi078.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-11T22:45:26.912 INFO:teuthology.orchestra.run.smithi078.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-11T22:45:26.946 INFO:teuthology.orchestra.run.smithi078.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-03-11T22:45:27.104 INFO:teuthology.orchestra.run.smithi078.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 141 kB/s | 22 kB 00:00 2024-03-11T22:45:27.163 INFO:teuthology.orchestra.run.smithi078.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 499 kB/s | 125 kB 00:00 2024-03-11T22:45:27.313 INFO:teuthology.orchestra.run.smithi078.stdout:(18/36): ceph-mon-19.0.0-1979.gb117a47a.el9.x86 3.4 MB/s | 4.7 MB 00:01 2024-03-11T22:45:27.330 INFO:teuthology.orchestra.run.smithi078.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 260 kB/s | 43 kB 00:00 2024-03-11T22:45:27.355 INFO:teuthology.orchestra.run.smithi078.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 603 kB/s | 151 kB 00:00 2024-03-11T22:45:27.405 INFO:teuthology.orchestra.run.smithi078.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 214 kB/s | 11 kB 00:00 2024-03-11T22:45:27.455 INFO:teuthology.orchestra.run.smithi078.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 353 kB/s | 18 kB 00:00 2024-03-11T22:45:27.506 INFO:teuthology.orchestra.run.smithi078.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 463 kB/s | 23 kB 00:00 2024-03-11T22:45:27.531 INFO:teuthology.orchestra.run.smithi078.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2024-03-11T22:45:27.556 INFO:teuthology.orchestra.run.smithi078.stdout:(25/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 706 kB/s | 172 kB 00:00 2024-03-11T22:45:27.573 INFO:teuthology.orchestra.run.smithi078.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 289 kB/s | 19 kB 00:00 2024-03-11T22:45:27.590 INFO:teuthology.orchestra.run.smithi078.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 334 kB/s | 20 kB 00:00 2024-03-11T22:45:27.607 INFO:teuthology.orchestra.run.smithi078.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 916 kB/s | 46 kB 00:00 2024-03-11T22:45:27.649 INFO:teuthology.orchestra.run.smithi078.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-03-11T22:45:27.666 INFO:teuthology.orchestra.run.smithi078.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 279 kB/s | 16 kB 00:00 2024-03-11T22:45:27.699 INFO:teuthology.orchestra.run.smithi078.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-03-11T22:45:27.717 INFO:teuthology.orchestra.run.smithi078.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 711 kB/s | 36 kB 00:00 2024-03-11T22:45:27.742 INFO:teuthology.orchestra.run.smithi078.stdout:(33/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 975 kB/s | 90 kB 00:00 2024-03-11T22:45:27.775 INFO:teuthology.orchestra.run.smithi078.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-11T22:45:27.801 INFO:teuthology.orchestra.run.smithi078.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 341 kB/s | 20 kB 00:00 2024-03-11T22:45:27.842 INFO:teuthology.orchestra.run.smithi078.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.3 MB/s | 427 kB 00:00 2024-03-11T22:45:27.845 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:45:27.846 INFO:teuthology.orchestra.run.smithi078.stdout:Total 11 MB/s | 30 MB 00:02 2024-03-11T22:45:27.869 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-03-11T22:45:27.910 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-03-11T22:45:27.910 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-03-11T22:45:28.320 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-03-11T22:45:28.320 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-03-11T22:45:29.657 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-03-11T22:45:29.720 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-11T22:45:29.752 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-11T22:45:29.780 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-11T22:45:29.851 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-11T22:45:29.905 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-11T22:45:29.946 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-11T22:45:30.029 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-11T22:45:30.132 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-11T22:45:30.174 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-11T22:45:30.245 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-11T22:45:30.299 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-11T22:45:30.337 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-11T22:45:30.386 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-11T22:45:30.450 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-11T22:45:30.634 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-11T22:45:30.684 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-11T22:45:30.722 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-11T22:45:30.756 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-11T22:45:30.787 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-11T22:45:30.863 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-11T22:45:30.897 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-11T22:45:30.930 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-11T22:45:30.970 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-11T22:45:31.005 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-11T22:45:31.011 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1979.gb117a47a.el9.x86_64 5/6 2024-03-11T22:45:31.040 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-11T22:45:31.085 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-test-2:19.0.0-1979.gb117a47a.el9.x86_64 6/6 2024-03-11T22:45:31.130 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-11T22:45:31.171 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-11T22:45:31.251 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-11T22:45:31.389 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-11T22:45:31.513 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-11T22:45:31.674 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1979.gb117a47a.el9. 31/36 2024-03-11T22:45:31.701 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-2:19.0.0-1979.gb117a47a.el9.x86_64 32/36 2024-03-11T22:45:31.726 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1979.gb117a47a.el9.x86_64 32/36 2024-03-11T22:45:31.726 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:45:31.726 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-11T22:45:31.726 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T22:45:31.726 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T22:45:31.727 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:45:33.182 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-osd-2:19.0.0-1979.gb117a47a.el9.x86_64 33/36 2024-03-11T22:45:33.208 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-osd-2:19.0.0-1979.gb117a47a.el9.x86_64 33/36 2024-03-11T22:45:33.208 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:45:33.208 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-11T22:45:33.208 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T22:45:33.208 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T22:45:33.208 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:45:33.474 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-test-2:19.0.0-1979.gb117a47a.el9.x86_64 6/6 2024-03-11T22:45:33.475 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-test-2:19.0.0-1979.gb117a47a.el9.x86_64 1/6 2024-03-11T22:45:33.475 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephsqlite-2:19.0.0-1979.gb117a47a.el9.x86_64 2/6 2024-03-11T22:45:33.475 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-11T22:45:33.475 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-11T22:45:33.475 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-11T22:45:33.652 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mon-2:19.0.0-1979.gb117a47a.el9.x86_64 34/36 2024-03-11T22:45:33.676 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mon-2:19.0.0-1979.gb117a47a.el9.x86_64 34/36 2024-03-11T22:45:33.676 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:45:33.676 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-11T22:45:33.676 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T22:45:33.676 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T22:45:33.676 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:45:33.841 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-11T22:45:33.841 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:45:33.841 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-11T22:45:33.841 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-test-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:33.841 INFO:teuthology.orchestra.run.smithi132.stdout: jq-1.6-16.el9.x86_64 2024-03-11T22:45:33.841 INFO:teuthology.orchestra.run.smithi132.stdout: libcephsqlite-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:33.841 INFO:teuthology.orchestra.run.smithi132.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-11T22:45:33.842 INFO:teuthology.orchestra.run.smithi132.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-11T22:45:33.842 INFO:teuthology.orchestra.run.smithi132.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-11T22:45:33.842 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:45:33.842 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-11T22:45:33.878 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mds-2:19.0.0-1979.gb117a47a.el9.x86_64 35/36 2024-03-11T22:45:33.901 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mds-2:19.0.0-1979.gb117a47a.el9.x86_64 35/36 2024-03-11T22:45:33.901 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:45:33.901 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-11T22:45:33.901 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T22:45:33.902 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T22:45:33.902 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:45:33.967 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-2:19.0.0-1979.gb117a47a.el9.x86_64 36/36 2024-03-11T22:45:34.159 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph 2024-03-11T22:45:34.740 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:00:41 ago on Mon 11 Mar 2024 10:44:53 PM UTC. 2024-03-11T22:45:34.901 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-11T22:45:34.904 INFO:teuthology.orchestra.run.smithi132.stdout:========================================================================================= 2024-03-11T22:45:34.904 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-03-11T22:45:34.904 INFO:teuthology.orchestra.run.smithi132.stdout:========================================================================================= 2024-03-11T22:45:34.904 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-11T22:45:34.904 INFO:teuthology.orchestra.run.smithi132.stdout: ceph x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 6.6 k 2024-03-11T22:45:34.904 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-03-11T22:45:34.904 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mds x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 2.1 M 2024-03-11T22:45:34.905 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 1.5 M 2024-03-11T22:45:34.905 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1979.gb117a47a.el9 ceph-noarch 248 k 2024-03-11T22:45:34.905 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mon x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 4.7 M 2024-03-11T22:45:34.905 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-osd x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 17 M 2024-03-11T22:45:34.905 INFO:teuthology.orchestra.run.smithi132.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-11T22:45:34.905 INFO:teuthology.orchestra.run.smithi132.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-11T22:45:34.905 INFO:teuthology.orchestra.run.smithi132.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-11T22:45:34.905 INFO:teuthology.orchestra.run.smithi132.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-11T22:45:34.905 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-11T22:45:34.905 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-11T22:45:34.905 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-11T22:45:34.905 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-11T22:45:34.906 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-11T22:45:34.906 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-11T22:45:34.906 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-11T22:45:34.906 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-11T22:45:34.906 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-11T22:45:34.906 INFO:teuthology.orchestra.run.smithi132.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-11T22:45:34.906 INFO:teuthology.orchestra.run.smithi132.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-11T22:45:34.906 INFO:teuthology.orchestra.run.smithi132.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-11T22:45:34.906 INFO:teuthology.orchestra.run.smithi132.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-11T22:45:34.906 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-11T22:45:34.906 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-11T22:45:34.906 INFO:teuthology.orchestra.run.smithi132.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-11T22:45:34.907 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-11T22:45:34.907 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-11T22:45:34.907 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-11T22:45:34.907 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-11T22:45:34.907 INFO:teuthology.orchestra.run.smithi132.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-11T22:45:34.907 INFO:teuthology.orchestra.run.smithi132.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-11T22:45:34.907 INFO:teuthology.orchestra.run.smithi132.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-11T22:45:34.907 INFO:teuthology.orchestra.run.smithi132.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-11T22:45:34.907 INFO:teuthology.orchestra.run.smithi132.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-11T22:45:34.907 INFO:teuthology.orchestra.run.smithi132.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-11T22:45:34.907 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:45:34.907 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-11T22:45:34.907 INFO:teuthology.orchestra.run.smithi132.stdout:========================================================================================= 2024-03-11T22:45:34.908 INFO:teuthology.orchestra.run.smithi132.stdout:Install 36 Packages 2024-03-11T22:45:34.908 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:45:34.908 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 30 M 2024-03-11T22:45:34.908 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 107 M 2024-03-11T22:45:34.908 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-11T22:45:35.418 INFO:teuthology.orchestra.run.smithi132.stdout:(1/36): ceph-19.0.0-1979.gb117a47a.el9.x86_64.r 41 kB/s | 6.6 kB 00:00 2024-03-11T22:45:35.756 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-2:19.0.0-1979.gb117a47a.el9.x86_64 36/36 2024-03-11T22:45:35.756 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-2:19.0.0-1979.gb117a47a.el9.x86_64 1/36 2024-03-11T22:45:35.756 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mds-2:19.0.0-1979.gb117a47a.el9.x86_64 2/36 2024-03-11T22:45:35.756 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-2:19.0.0-1979.gb117a47a.el9.x86_64 3/36 2024-03-11T22:45:35.756 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mon-2:19.0.0-1979.gb117a47a.el9.x86_64 4/36 2024-03-11T22:45:35.757 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-osd-2:19.0.0-1979.gb117a47a.el9.x86_64 5/36 2024-03-11T22:45:35.757 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1979.gb117a47a.el9. 6/36 2024-03-11T22:45:35.757 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-11T22:45:35.757 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-11T22:45:35.757 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-11T22:45:35.757 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-11T22:45:35.757 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-11T22:45:35.757 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-11T22:45:35.757 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-11T22:45:35.757 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-11T22:45:35.757 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-11T22:45:35.757 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-11T22:45:35.757 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-11T22:45:35.757 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-11T22:45:35.758 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-11T22:45:35.758 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-11T22:45:35.758 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-11T22:45:35.758 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-11T22:45:35.758 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-11T22:45:35.758 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-11T22:45:35.758 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-11T22:45:35.758 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-11T22:45:35.758 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-11T22:45:35.758 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-11T22:45:35.758 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-11T22:45:35.758 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-11T22:45:35.758 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-11T22:45:35.759 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-11T22:45:35.759 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-11T22:45:35.759 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-11T22:45:35.759 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-11T22:45:35.843 INFO:teuthology.orchestra.run.smithi132.stdout:(2/36): ceph-mon-19.0.0-1979.gb117a47a.el9.x86_ 11 MB/s | 4.7 MB 00:00 2024-03-11T22:45:35.894 INFO:teuthology.orchestra.run.smithi132.stdout:(3/36): ceph-mgr-19.0.0-1979.gb117a47a.el9.x86_ 2.3 MB/s | 1.5 MB 00:00 2024-03-11T22:45:36.011 INFO:teuthology.orchestra.run.smithi132.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1979.gb117 2.1 MB/s | 248 kB 00:00 2024-03-11T22:45:36.094 INFO:teuthology.orchestra.run.smithi132.stdout:(5/36): ceph-mds-19.0.0-1979.gb117a47a.el9.x86_ 2.5 MB/s | 2.1 MB 00:00 2024-03-11T22:45:36.178 INFO:teuthology.orchestra.run.smithi132.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-03-11T22:45:36.203 INFO:teuthology.orchestra.run.smithi132.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 144 kB 00:00 2024-03-11T22:45:36.228 INFO:teuthology.orchestra.run.smithi132.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 632 kB/s | 32 kB 00:00 2024-03-11T22:45:36.254 INFO:teuthology.orchestra.run.smithi132.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 941 kB/s | 47 kB 00:00 2024-03-11T22:45:36.279 INFO:teuthology.orchestra.run.smithi132.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 769 kB/s | 39 kB 00:00 2024-03-11T22:45:36.304 INFO:teuthology.orchestra.run.smithi132.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-11T22:45:36.363 INFO:teuthology.orchestra.run.smithi132.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-11T22:45:36.388 INFO:teuthology.orchestra.run.smithi132.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-11T22:45:36.413 INFO:teuthology.orchestra.run.smithi132.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-11T22:45:36.447 INFO:teuthology.orchestra.run.smithi132.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.1 MB/s | 187 kB 00:00 2024-03-11T22:45:36.673 INFO:teuthology.orchestra.run.smithi132.stdout:(16/36): ceph-osd-19.0.0-1979.gb117a47a.el9.x86 20 MB/s | 17 MB 00:00 2024-03-11T22:45:36.698 INFO:teuthology.orchestra.run.smithi132.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 89 kB/s | 22 kB 00:00 2024-03-11T22:45:36.748 INFO:teuthology.orchestra.run.smithi132.stdout:(18/36): python3-requests-2.25.1-8.el9.noarch.r 374 kB/s | 125 kB 00:00 2024-03-11T22:45:36.788 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-11T22:45:36.789 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:45:36.789 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-03-11T22:45:36.789 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:36.789 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mds-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:36.789 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:36.789 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-modules-core-2:19.0.0-1979.gb117a47a.el9.noarch 2024-03-11T22:45:36.789 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mon-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:36.789 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-osd-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:36.789 INFO:teuthology.orchestra.run.smithi078.stdout: lua-5.4.4-4.el9.x86_64 2024-03-11T22:45:36.789 INFO:teuthology.orchestra.run.smithi078.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-11T22:45:36.789 INFO:teuthology.orchestra.run.smithi078.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-11T22:45:36.790 INFO:teuthology.orchestra.run.smithi078.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-11T22:45:36.790 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-11T22:45:36.790 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-11T22:45:36.790 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-11T22:45:36.790 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-11T22:45:36.790 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-11T22:45:36.790 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-11T22:45:36.790 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-11T22:45:36.790 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-11T22:45:36.790 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-11T22:45:36.790 INFO:teuthology.orchestra.run.smithi078.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-11T22:45:36.790 INFO:teuthology.orchestra.run.smithi078.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-11T22:45:36.791 INFO:teuthology.orchestra.run.smithi078.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-11T22:45:36.791 INFO:teuthology.orchestra.run.smithi078.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-11T22:45:36.791 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-11T22:45:36.791 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ply-3.11-14.el9.noarch 2024-03-11T22:45:36.791 INFO:teuthology.orchestra.run.smithi078.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-11T22:45:36.791 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-11T22:45:36.791 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-11T22:45:36.791 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-11T22:45:36.791 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-11T22:45:36.791 INFO:teuthology.orchestra.run.smithi078.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-11T22:45:36.791 INFO:teuthology.orchestra.run.smithi078.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-11T22:45:36.791 INFO:teuthology.orchestra.run.smithi078.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-11T22:45:36.792 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-11T22:45:36.792 INFO:teuthology.orchestra.run.smithi078.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-11T22:45:36.792 INFO:teuthology.orchestra.run.smithi078.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-11T22:45:36.792 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:45:36.792 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-03-11T22:45:37.157 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-base 2024-03-11T22:45:37.248 INFO:teuthology.orchestra.run.smithi132.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 79 kB/s | 43 kB 00:00 2024-03-11T22:45:37.725 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:00:55 ago on Mon 11 Mar 2024 10:44:42 PM UTC. 2024-03-11T22:45:37.790 INFO:teuthology.orchestra.run.smithi132.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 135 kB/s | 151 kB 00:01 2024-03-11T22:45:37.822 INFO:teuthology.orchestra.run.smithi078.stdout:Package ceph-base-2:19.0.0-1979.gb117a47a.el9.x86_64 is already installed. 2024-03-11T22:45:37.873 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-03-11T22:45:37.874 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2024-03-11T22:45:37.874 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-03-11T22:45:37.899 INFO:teuthology.orchestra.run.smithi132.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 149 kB/s | 172 kB 00:01 2024-03-11T22:45:37.924 INFO:teuthology.orchestra.run.smithi132.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 80 kB/s | 11 kB 00:00 2024-03-11T22:45:37.943 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install cephadm 2024-03-11T22:45:38.016 INFO:teuthology.orchestra.run.smithi132.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 151 kB/s | 18 kB 00:00 2024-03-11T22:45:38.133 INFO:teuthology.orchestra.run.smithi132.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 111 kB/s | 23 kB 00:00 2024-03-11T22:45:38.150 INFO:teuthology.orchestra.run.smithi132.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 145 kB/s | 19 kB 00:00 2024-03-11T22:45:38.242 INFO:teuthology.orchestra.run.smithi132.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 181 kB/s | 20 kB 00:00 2024-03-11T22:45:38.450 INFO:teuthology.orchestra.run.smithi132.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 154 kB/s | 46 kB 00:00 2024-03-11T22:45:38.505 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:00:56 ago on Mon 11 Mar 2024 10:44:42 PM UTC. 2024-03-11T22:45:38.634 INFO:teuthology.orchestra.run.smithi132.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 201 kB/s | 79 kB 00:00 2024-03-11T22:45:38.654 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-03-11T22:45:38.657 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-11T22:45:38.657 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2024-03-11T22:45:38.657 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-11T22:45:38.658 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-03-11T22:45:38.658 INFO:teuthology.orchestra.run.smithi078.stdout: cephadm noarch 2:19.0.0-1979.gb117a47a.el9 ceph-noarch 748 k 2024-03-11T22:45:38.658 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:45:38.658 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-03-11T22:45:38.658 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-11T22:45:38.658 INFO:teuthology.orchestra.run.smithi078.stdout:Install 1 Package 2024-03-11T22:45:38.658 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:45:38.658 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 748 k 2024-03-11T22:45:38.659 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 755 k 2024-03-11T22:45:38.659 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-03-11T22:45:38.759 INFO:teuthology.orchestra.run.smithi132.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 132 kB/s | 16 kB 00:00 2024-03-11T22:45:38.843 INFO:teuthology.orchestra.run.smithi132.stdout:(30/36): python3-cherrypy-18.6.1-2.el9.noarch.r 225 kB/s | 358 kB 00:01 2024-03-11T22:45:39.060 INFO:teuthology.orchestra.run.smithi132.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 165 kB/s | 36 kB 00:00 2024-03-11T22:45:39.077 INFO:teuthology.orchestra.run.smithi078.stdout:cephadm-19.0.0-1979.gb117a47a.el9.noarch.rpm 1.7 MB/s | 748 kB 00:00 2024-03-11T22:45:39.078 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:45:39.078 INFO:teuthology.orchestra.run.smithi078.stdout:Total 1.7 MB/s | 748 kB 00:00 2024-03-11T22:45:39.078 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-03-11T22:45:39.083 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-03-11T22:45:39.083 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-03-11T22:45:39.089 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-03-11T22:45:39.089 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-03-11T22:45:39.163 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-03-11T22:45:39.177 INFO:teuthology.orchestra.run.smithi132.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 215 kB/s | 90 kB 00:00 2024-03-11T22:45:39.460 INFO:teuthology.orchestra.run.smithi132.stdout:(33/36): python3-pecan-1.4.2-3.el9.noarch.rpm 270 kB/s | 272 kB 00:01 2024-03-11T22:45:39.586 INFO:teuthology.orchestra.run.smithi132.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 160 kB/s | 20 kB 00:00 2024-03-11T22:45:39.596 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: cephadm-2:19.0.0-1979.gb117a47a.el9.noarch 1/1 2024-03-11T22:45:39.669 INFO:teuthology.orchestra.run.smithi132.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 377 kB/s | 230 kB 00:00 2024-03-11T22:45:39.681 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cephadm-2:19.0.0-1979.gb117a47a.el9.noarch 1/1 2024-03-11T22:45:40.099 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: cephadm-2:19.0.0-1979.gb117a47a.el9.noarch 1/1 2024-03-11T22:45:40.236 INFO:teuthology.orchestra.run.smithi132.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 403 kB/s | 427 kB 00:01 2024-03-11T22:45:40.238 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:45:40.239 INFO:teuthology.orchestra.run.smithi132.stdout:Total 5.6 MB/s | 30 MB 00:05 2024-03-11T22:45:40.264 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-11T22:45:40.306 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-11T22:45:40.306 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-11T22:45:40.397 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cephadm-2:19.0.0-1979.gb117a47a.el9.noarch 1/1 2024-03-11T22:45:40.397 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:45:40.397 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-03-11T22:45:40.397 INFO:teuthology.orchestra.run.smithi078.stdout: cephadm-2:19.0.0-1979.gb117a47a.el9.noarch 2024-03-11T22:45:40.397 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:45:40.397 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-03-11T22:45:40.598 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-immutable-object-cache 2024-03-11T22:45:40.723 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-11T22:45:40.723 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-11T22:45:41.162 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:00:59 ago on Mon 11 Mar 2024 10:44:42 PM UTC. 2024-03-11T22:45:41.314 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-03-11T22:45:41.315 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-11T22:45:41.315 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repo Size 2024-03-11T22:45:41.315 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-11T22:45:41.315 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-03-11T22:45:41.315 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 145 k 2024-03-11T22:45:41.316 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:45:41.316 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-03-11T22:45:41.316 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-11T22:45:41.316 INFO:teuthology.orchestra.run.smithi078.stdout:Install 1 Package 2024-03-11T22:45:41.316 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:45:41.316 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 145 k 2024-03-11T22:45:41.316 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 436 k 2024-03-11T22:45:41.316 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-03-11T22:45:41.717 INFO:teuthology.orchestra.run.smithi078.stdout:ceph-immutable-object-cache-19.0.0-1979.gb117a4 363 kB/s | 145 kB 00:00 2024-03-11T22:45:41.717 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:45:41.717 INFO:teuthology.orchestra.run.smithi078.stdout:Total 362 kB/s | 145 kB 00:00 2024-03-11T22:45:41.717 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-03-11T22:45:41.724 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-03-11T22:45:41.724 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-03-11T22:45:41.777 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-03-11T22:45:41.777 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-03-11T22:45:41.925 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-03-11T22:45:41.983 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1979.gb117a47a. 1/1 2024-03-11T22:45:42.008 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1979.gb117a47a. 1/1 2024-03-11T22:45:42.008 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:45:42.008 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-11T22:45:42.008 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:45:42.044 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-11T22:45:42.105 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-11T22:45:42.138 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-11T22:45:42.166 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-11T22:45:42.236 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-11T22:45:42.281 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-11T22:45:42.314 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-11T22:45:42.404 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-11T22:45:42.484 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-11T22:45:42.525 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-11T22:45:42.596 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-11T22:45:42.650 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-11T22:45:42.689 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-11T22:45:42.746 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-11T22:45:42.801 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-11T22:45:42.921 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1979.gb117a47a. 1/1 2024-03-11T22:45:42.922 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:45:42.922 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-03-11T22:45:42.922 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-immutable-object-cache-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:42.922 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:45:42.922 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-03-11T22:45:42.994 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-11T22:45:43.044 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-11T22:45:43.082 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-11T22:45:43.125 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-11T22:45:43.156 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-11T22:45:43.162 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-mgr 2024-03-11T22:45:43.232 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-11T22:45:43.274 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-11T22:45:43.308 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-11T22:45:43.348 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-11T22:45:43.383 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-11T22:45:43.418 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-11T22:45:43.507 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-11T22:45:43.540 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-11T22:45:43.612 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-11T22:45:43.733 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:01 ago on Mon 11 Mar 2024 10:44:42 PM UTC. 2024-03-11T22:45:43.756 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-11T22:45:43.830 INFO:teuthology.orchestra.run.smithi078.stdout:Package ceph-mgr-2:19.0.0-1979.gb117a47a.el9.x86_64 is already installed. 2024-03-11T22:45:43.881 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-11T22:45:43.882 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-03-11T22:45:43.883 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2024-03-11T22:45:43.883 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-03-11T22:45:43.951 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-mgr-dashboard 2024-03-11T22:45:44.044 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1979.gb117a47a.el9. 31/36 2024-03-11T22:45:44.070 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-2:19.0.0-1979.gb117a47a.el9.x86_64 32/36 2024-03-11T22:45:44.096 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1979.gb117a47a.el9.x86_64 32/36 2024-03-11T22:45:44.096 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:45:44.096 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-11T22:45:44.096 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T22:45:44.096 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T22:45:44.096 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:45:44.508 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:02 ago on Mon 11 Mar 2024 10:44:42 PM UTC. 2024-03-11T22:45:44.658 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-03-11T22:45:44.659 INFO:teuthology.orchestra.run.smithi078.stdout:====================================================================================== 2024-03-11T22:45:44.659 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2024-03-11T22:45:44.659 INFO:teuthology.orchestra.run.smithi078.stdout:====================================================================================== 2024-03-11T22:45:44.659 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-03-11T22:45:44.659 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1979.gb117a47a.el9 ceph-noarch 1.8 M 2024-03-11T22:45:44.659 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-03-11T22:45:44.659 INFO:teuthology.orchestra.run.smithi078.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-11T22:45:44.659 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1979.gb117a47a.el9 ceph-noarch 34 k 2024-03-11T22:45:44.659 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1979.gb117a47a.el9 ceph-noarch 16 k 2024-03-11T22:45:44.659 INFO:teuthology.orchestra.run.smithi078.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-11T22:45:44.659 INFO:teuthology.orchestra.run.smithi078.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-11T22:45:44.659 INFO:teuthology.orchestra.run.smithi078.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-11T22:45:44.660 INFO:teuthology.orchestra.run.smithi078.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-11T22:45:44.660 INFO:teuthology.orchestra.run.smithi078.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-11T22:45:44.660 INFO:teuthology.orchestra.run.smithi078.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-11T22:45:44.660 INFO:teuthology.orchestra.run.smithi078.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-11T22:45:44.660 INFO:teuthology.orchestra.run.smithi078.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-11T22:45:44.660 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:45:44.660 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-03-11T22:45:44.660 INFO:teuthology.orchestra.run.smithi078.stdout:====================================================================================== 2024-03-11T22:45:44.660 INFO:teuthology.orchestra.run.smithi078.stdout:Install 12 Packages 2024-03-11T22:45:44.660 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:45:44.661 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 6.9 M 2024-03-11T22:45:44.661 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 26 M 2024-03-11T22:45:44.661 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-03-11T22:45:45.178 INFO:teuthology.orchestra.run.smithi078.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1979.gb1 80 kB/s | 34 kB 00:00 2024-03-11T22:45:45.312 INFO:teuthology.orchestra.run.smithi078.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1979.gb11 29 kB/s | 16 kB 00:00 2024-03-11T22:45:45.362 INFO:teuthology.orchestra.run.smithi078.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.5 MB/s | 1.0 MB 00:00 2024-03-11T22:45:45.387 INFO:teuthology.orchestra.run.smithi078.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 3.1 MB/s | 238 kB 00:00 2024-03-11T22:45:45.496 INFO:teuthology.orchestra.run.smithi078.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 4.9 MB/s | 548 kB 00:00 2024-03-11T22:45:45.529 INFO:teuthology.orchestra.run.smithi078.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 582 kB/s | 19 kB 00:00 2024-03-11T22:45:45.530 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-osd-2:19.0.0-1979.gb117a47a.el9.x86_64 33/36 2024-03-11T22:45:45.559 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-osd-2:19.0.0-1979.gb117a47a.el9.x86_64 33/36 2024-03-11T22:45:45.559 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:45:45.560 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-11T22:45:45.560 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T22:45:45.560 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T22:45:45.560 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:45:45.613 INFO:teuthology.orchestra.run.smithi078.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 25 MB/s | 2.0 MB 00:00 2024-03-11T22:45:45.655 INFO:teuthology.orchestra.run.smithi078.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 3.4 MB/s | 144 kB 00:00 2024-03-11T22:45:45.689 INFO:teuthology.orchestra.run.smithi078.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 929 kB/s | 31 kB 00:00 2024-03-11T22:45:45.731 INFO:teuthology.orchestra.run.smithi078.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 4.4 MB/s | 188 kB 00:00 2024-03-11T22:45:45.806 INFO:teuthology.orchestra.run.smithi078.stdout:(11/12): ceph-mgr-dashboard-19.0.0-1979.gb117a4 1.7 MB/s | 1.8 MB 00:01 2024-03-11T22:45:45.997 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mon-2:19.0.0-1979.gb117a47a.el9.x86_64 34/36 2024-03-11T22:45:46.020 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mon-2:19.0.0-1979.gb117a47a.el9.x86_64 34/36 2024-03-11T22:45:46.020 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:45:46.020 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-11T22:45:46.020 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T22:45:46.020 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T22:45:46.020 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:45:46.239 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mds-2:19.0.0-1979.gb117a47a.el9.x86_64 35/36 2024-03-11T22:45:46.262 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mds-2:19.0.0-1979.gb117a47a.el9.x86_64 35/36 2024-03-11T22:45:46.262 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:45:46.262 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-11T22:45:46.263 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T22:45:46.263 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T22:45:46.263 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:45:46.328 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-2:19.0.0-1979.gb117a47a.el9.x86_64 36/36 2024-03-11T22:45:46.473 INFO:teuthology.orchestra.run.smithi078.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 778 kB/s | 863 kB 00:01 2024-03-11T22:45:46.476 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:45:46.476 INFO:teuthology.orchestra.run.smithi078.stdout:Total 3.8 MB/s | 6.9 MB 00:01 2024-03-11T22:45:46.496 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-03-11T22:45:46.515 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-03-11T22:45:46.515 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-03-11T22:45:46.690 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-03-11T22:45:46.690 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-03-11T22:45:47.204 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-03-11T22:45:47.265 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-11T22:45:47.300 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-11T22:45:47.346 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-11T22:45:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-11T22:45:47.492 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-11T22:45:47.618 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-11T22:45:47.713 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-11T22:45:47.757 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-11T22:45:47.810 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-11T22:45:47.842 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1979.gb117a47a.el9 10/12 2024-03-11T22:45:48.079 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1979.gb117a47a.el 11/12 2024-03-11T22:45:48.088 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-2:19.0.0-1979.gb117a47a.el9.x86_64 36/36 2024-03-11T22:45:48.089 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-2:19.0.0-1979.gb117a47a.el9.x86_64 1/36 2024-03-11T22:45:48.089 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mds-2:19.0.0-1979.gb117a47a.el9.x86_64 2/36 2024-03-11T22:45:48.089 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-2:19.0.0-1979.gb117a47a.el9.x86_64 3/36 2024-03-11T22:45:48.089 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mon-2:19.0.0-1979.gb117a47a.el9.x86_64 4/36 2024-03-11T22:45:48.089 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-osd-2:19.0.0-1979.gb117a47a.el9.x86_64 5/36 2024-03-11T22:45:48.089 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1979.gb117a47a.el9. 6/36 2024-03-11T22:45:48.089 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-11T22:45:48.089 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-11T22:45:48.089 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-11T22:45:48.089 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-11T22:45:48.089 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-11T22:45:48.089 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-11T22:45:48.090 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-11T22:45:48.090 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-11T22:45:48.090 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-11T22:45:48.090 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-11T22:45:48.090 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-11T22:45:48.090 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-11T22:45:48.090 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-11T22:45:48.090 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-11T22:45:48.090 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-11T22:45:48.090 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-11T22:45:48.090 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-11T22:45:48.090 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-11T22:45:48.090 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-11T22:45:48.091 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-11T22:45:48.091 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-11T22:45:48.091 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-11T22:45:48.091 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-11T22:45:48.091 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-11T22:45:48.091 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-11T22:45:48.091 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-11T22:45:48.091 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-11T22:45:48.091 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-11T22:45:48.091 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-11T22:45:48.108 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1979.gb117a47a.el9.noa 12/12 2024-03-11T22:45:48.902 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1979.gb117a47a.el9.noa 12/12 2024-03-11T22:45:48.902 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1979.gb117a47a.el 1/12 2024-03-11T22:45:48.902 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1979.gb117a47a.el9.noa 2/12 2024-03-11T22:45:48.902 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1979.gb117a47a.el9 3/12 2024-03-11T22:45:48.902 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-11T22:45:48.902 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-11T22:45:48.902 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-11T22:45:48.902 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-11T22:45:48.902 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-11T22:45:48.902 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-11T22:45:48.902 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-11T22:45:48.903 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-11T22:45:49.259 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-11T22:45:49.259 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:45:49.259 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-11T22:45:49.259 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:49.259 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mds-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:49.259 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:49.260 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-modules-core-2:19.0.0-1979.gb117a47a.el9.noarch 2024-03-11T22:45:49.260 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mon-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:49.260 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-osd-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:49.260 INFO:teuthology.orchestra.run.smithi132.stdout: lua-5.4.4-4.el9.x86_64 2024-03-11T22:45:49.260 INFO:teuthology.orchestra.run.smithi132.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-11T22:45:49.260 INFO:teuthology.orchestra.run.smithi132.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-11T22:45:49.260 INFO:teuthology.orchestra.run.smithi132.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-11T22:45:49.260 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-11T22:45:49.260 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-11T22:45:49.260 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-11T22:45:49.260 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-11T22:45:49.260 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-11T22:45:49.261 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-11T22:45:49.261 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-11T22:45:49.261 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-11T22:45:49.261 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-11T22:45:49.261 INFO:teuthology.orchestra.run.smithi132.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-11T22:45:49.261 INFO:teuthology.orchestra.run.smithi132.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-11T22:45:49.261 INFO:teuthology.orchestra.run.smithi132.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-11T22:45:49.261 INFO:teuthology.orchestra.run.smithi132.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-11T22:45:49.261 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-11T22:45:49.261 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ply-3.11-14.el9.noarch 2024-03-11T22:45:49.261 INFO:teuthology.orchestra.run.smithi132.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-11T22:45:49.261 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-11T22:45:49.261 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-11T22:45:49.261 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-11T22:45:49.262 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-11T22:45:49.262 INFO:teuthology.orchestra.run.smithi132.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-11T22:45:49.262 INFO:teuthology.orchestra.run.smithi132.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-11T22:45:49.262 INFO:teuthology.orchestra.run.smithi132.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-11T22:45:49.262 INFO:teuthology.orchestra.run.smithi132.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-11T22:45:49.262 INFO:teuthology.orchestra.run.smithi132.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-11T22:45:49.262 INFO:teuthology.orchestra.run.smithi132.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-11T22:45:49.262 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:45:49.262 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-11T22:45:49.270 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-11T22:45:49.270 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:45:49.270 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-03-11T22:45:49.270 INFO:teuthology.orchestra.run.smithi078.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-11T22:45:49.270 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-grafana-dashboards-2:19.0.0-1979.gb117a47a.el9.noarch 2024-03-11T22:45:49.270 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-dashboard-2:19.0.0-1979.gb117a47a.el9.noarch 2024-03-11T22:45:49.270 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-prometheus-alerts-2:19.0.0-1979.gb117a47a.el9.noarch 2024-03-11T22:45:49.270 INFO:teuthology.orchestra.run.smithi078.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-11T22:45:49.270 INFO:teuthology.orchestra.run.smithi078.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-11T22:45:49.271 INFO:teuthology.orchestra.run.smithi078.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-11T22:45:49.271 INFO:teuthology.orchestra.run.smithi078.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-11T22:45:49.271 INFO:teuthology.orchestra.run.smithi078.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-11T22:45:49.271 INFO:teuthology.orchestra.run.smithi078.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-11T22:45:49.271 INFO:teuthology.orchestra.run.smithi078.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-11T22:45:49.271 INFO:teuthology.orchestra.run.smithi078.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-11T22:45:49.271 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:45:49.271 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-03-11T22:45:49.538 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-11T22:45:49.625 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-base 2024-03-11T22:45:50.097 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:08 ago on Mon 11 Mar 2024 10:44:42 PM UTC. 2024-03-11T22:45:50.188 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:00:57 ago on Mon 11 Mar 2024 10:44:53 PM UTC. 2024-03-11T22:45:50.247 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-03-11T22:45:50.249 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================================ 2024-03-11T22:45:50.249 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2024-03-11T22:45:50.249 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================================ 2024-03-11T22:45:50.249 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-03-11T22:45:50.249 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1979.gb117a47a.el9 ceph-noarch 7.4 M 2024-03-11T22:45:50.249 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-03-11T22:45:50.249 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-11T22:45:50.249 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-11T22:45:50.249 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-11T22:45:50.249 INFO:teuthology.orchestra.run.smithi078.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-11T22:45:50.249 INFO:teuthology.orchestra.run.smithi078.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-11T22:45:50.249 INFO:teuthology.orchestra.run.smithi078.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-11T22:45:50.249 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-11T22:45:50.250 INFO:teuthology.orchestra.run.smithi078.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-11T22:45:50.250 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-11T22:45:50.250 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-11T22:45:50.250 INFO:teuthology.orchestra.run.smithi078.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-11T22:45:50.250 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:45:50.250 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-03-11T22:45:50.250 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================================ 2024-03-11T22:45:50.250 INFO:teuthology.orchestra.run.smithi078.stdout:Install 12 Packages 2024-03-11T22:45:50.250 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:45:50.251 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 37 M 2024-03-11T22:45:50.251 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 209 M 2024-03-11T22:45:50.251 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-03-11T22:45:50.285 INFO:teuthology.orchestra.run.smithi132.stdout:Package ceph-base-2:19.0.0-1979.gb117a47a.el9.x86_64 is already installed. 2024-03-11T22:45:50.336 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-11T22:45:50.337 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-03-11T22:45:50.337 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-11T22:45:50.397 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install cephadm 2024-03-11T22:45:50.871 INFO:teuthology.orchestra.run.smithi078.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 203 kB/s | 32 kB 00:00 2024-03-11T22:45:50.913 INFO:teuthology.orchestra.run.smithi078.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 416 kB/s | 17 kB 00:00 2024-03-11T22:45:50.976 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:00:57 ago on Mon 11 Mar 2024 10:44:53 PM UTC. 2024-03-11T22:45:50.979 INFO:teuthology.orchestra.run.smithi078.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-11T22:45:50.997 INFO:teuthology.orchestra.run.smithi078.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 414 kB/s | 34 kB 00:00 2024-03-11T22:45:51.123 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-11T22:45:51.124 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-11T22:45:51.124 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-03-11T22:45:51.124 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-11T22:45:51.124 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-11T22:45:51.124 INFO:teuthology.orchestra.run.smithi132.stdout: cephadm noarch 2:19.0.0-1979.gb117a47a.el9 ceph-noarch 748 k 2024-03-11T22:45:51.124 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:45:51.124 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-11T22:45:51.124 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-11T22:45:51.124 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-11T22:45:51.124 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:45:51.125 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 748 k 2024-03-11T22:45:51.125 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 755 k 2024-03-11T22:45:51.125 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-11T22:45:51.155 INFO:teuthology.orchestra.run.smithi078.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2024-03-11T22:45:51.230 INFO:teuthology.orchestra.run.smithi078.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-11T22:45:51.256 INFO:teuthology.orchestra.run.smithi078.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-03-11T22:45:51.473 INFO:teuthology.orchestra.run.smithi078.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 65 MB/s | 16 MB 00:00 2024-03-11T22:45:51.573 INFO:teuthology.orchestra.run.smithi078.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-19 8.6 MB/s | 7.4 MB 00:00 2024-03-11T22:45:51.656 INFO:teuthology.orchestra.run.smithi078.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.0 MB/s | 803 kB 00:00 2024-03-11T22:45:51.815 INFO:teuthology.orchestra.run.smithi078.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 571 kB/s | 195 kB 00:00 2024-03-11T22:45:52.059 INFO:teuthology.orchestra.run.smithi132.stdout:cephadm-19.0.0-1979.gb117a47a.el9.noarch.rpm 801 kB/s | 748 kB 00:00 2024-03-11T22:45:52.060 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:45:52.060 INFO:teuthology.orchestra.run.smithi132.stdout:Total 800 kB/s | 748 kB 00:00 2024-03-11T22:45:52.060 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-11T22:45:52.066 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-11T22:45:52.066 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-11T22:45:52.071 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-11T22:45:52.072 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-11T22:45:52.149 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-11T22:45:52.604 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: cephadm-2:19.0.0-1979.gb117a47a.el9.noarch 1/1 2024-03-11T22:45:52.688 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : cephadm-2:19.0.0-1979.gb117a47a.el9.noarch 1/1 2024-03-11T22:45:52.891 INFO:teuthology.orchestra.run.smithi078.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 186 kB/s | 245 kB 00:01 2024-03-11T22:45:52.894 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:45:52.894 INFO:teuthology.orchestra.run.smithi078.stdout:Total 14 MB/s | 37 MB 00:02 2024-03-11T22:45:52.938 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-03-11T22:45:52.965 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-03-11T22:45:52.965 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-03-11T22:45:53.098 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: cephadm-2:19.0.0-1979.gb117a47a.el9.noarch 1/1 2024-03-11T22:45:53.342 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-03-11T22:45:53.342 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-03-11T22:45:53.405 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : cephadm-2:19.0.0-1979.gb117a47a.el9.noarch 1/1 2024-03-11T22:45:53.405 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:45:53.405 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-11T22:45:53.405 INFO:teuthology.orchestra.run.smithi132.stdout: cephadm-2:19.0.0-1979.gb117a47a.el9.noarch 2024-03-11T22:45:53.405 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:45:53.405 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-11T22:45:53.622 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-immutable-object-cache 2024-03-11T22:45:54.041 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-03-11T22:45:54.113 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-11T22:45:54.152 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-11T22:45:54.190 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:01 ago on Mon 11 Mar 2024 10:44:53 PM UTC. 2024-03-11T22:45:54.246 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-11T22:45:54.281 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-11T22:45:54.343 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-11T22:45:54.344 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-11T22:45:54.344 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-03-11T22:45:54.344 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-11T22:45:54.345 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-11T22:45:54.345 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 145 k 2024-03-11T22:45:54.345 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:45:54.345 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-11T22:45:54.345 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-11T22:45:54.345 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-11T22:45:54.345 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:45:54.345 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 145 k 2024-03-11T22:45:54.345 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 436 k 2024-03-11T22:45:54.345 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-11T22:45:54.452 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-11T22:45:54.480 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-11T22:45:54.565 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-11T22:45:54.791 INFO:teuthology.orchestra.run.smithi132.stdout:ceph-immutable-object-cache-19.0.0-1979.gb117a4 326 kB/s | 145 kB 00:00 2024-03-11T22:45:54.791 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:45:54.791 INFO:teuthology.orchestra.run.smithi132.stdout:Total 325 kB/s | 145 kB 00:00 2024-03-11T22:45:54.792 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-11T22:45:54.799 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-11T22:45:54.799 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-11T22:45:54.851 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-11T22:45:54.852 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-11T22:45:55.004 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-11T22:45:55.066 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1979.gb117a47a. 1/1 2024-03-11T22:45:55.091 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1979.gb117a47a. 1/1 2024-03-11T22:45:55.091 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:45:55.091 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-11T22:45:55.091 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:45:55.196 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-11T22:45:55.299 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-11T22:45:56.032 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1979.gb117a47a. 1/1 2024-03-11T22:45:56.032 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:45:56.032 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-11T22:45:56.032 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-immutable-object-cache-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:45:56.032 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:45:56.032 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-11T22:45:56.280 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr 2024-03-11T22:45:56.533 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-11T22:45:56.861 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:03 ago on Mon 11 Mar 2024 10:44:53 PM UTC. 2024-03-11T22:45:56.958 INFO:teuthology.orchestra.run.smithi132.stdout:Package ceph-mgr-2:19.0.0-1979.gb117a47a.el9.x86_64 is already installed. 2024-03-11T22:45:57.009 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-11T22:45:57.010 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-03-11T22:45:57.010 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-11T22:45:57.081 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr-dashboard 2024-03-11T22:45:57.566 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-11T22:45:57.587 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1979.gb117a 12/12 2024-03-11T22:45:57.643 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:04 ago on Mon 11 Mar 2024 10:44:53 PM UTC. 2024-03-11T22:45:57.794 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-11T22:45:57.795 INFO:teuthology.orchestra.run.smithi132.stdout:====================================================================================== 2024-03-11T22:45:57.795 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-03-11T22:45:57.795 INFO:teuthology.orchestra.run.smithi132.stdout:====================================================================================== 2024-03-11T22:45:57.795 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-11T22:45:57.795 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1979.gb117a47a.el9 ceph-noarch 1.8 M 2024-03-11T22:45:57.795 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-03-11T22:45:57.795 INFO:teuthology.orchestra.run.smithi132.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-11T22:45:57.795 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1979.gb117a47a.el9 ceph-noarch 34 k 2024-03-11T22:45:57.795 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1979.gb117a47a.el9 ceph-noarch 16 k 2024-03-11T22:45:57.795 INFO:teuthology.orchestra.run.smithi132.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-11T22:45:57.795 INFO:teuthology.orchestra.run.smithi132.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-11T22:45:57.796 INFO:teuthology.orchestra.run.smithi132.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-11T22:45:57.796 INFO:teuthology.orchestra.run.smithi132.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-11T22:45:57.796 INFO:teuthology.orchestra.run.smithi132.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-11T22:45:57.796 INFO:teuthology.orchestra.run.smithi132.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-11T22:45:57.796 INFO:teuthology.orchestra.run.smithi132.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-11T22:45:57.796 INFO:teuthology.orchestra.run.smithi132.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-11T22:45:57.796 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:45:57.796 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-11T22:45:57.796 INFO:teuthology.orchestra.run.smithi132.stdout:====================================================================================== 2024-03-11T22:45:57.796 INFO:teuthology.orchestra.run.smithi132.stdout:Install 12 Packages 2024-03-11T22:45:57.796 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:45:57.797 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 6.9 M 2024-03-11T22:45:57.797 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 26 M 2024-03-11T22:45:57.797 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-11T22:45:58.378 INFO:teuthology.orchestra.run.smithi132.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1979.gb11 86 kB/s | 16 kB 00:00 2024-03-11T22:45:58.495 INFO:teuthology.orchestra.run.smithi132.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1979.gb1 110 kB/s | 34 kB 00:00 2024-03-11T22:45:58.562 INFO:teuthology.orchestra.run.smithi132.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.5 MB/s | 1.0 MB 00:00 2024-03-11T22:45:58.587 INFO:teuthology.orchestra.run.smithi132.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.5 MB/s | 238 kB 00:00 2024-03-11T22:45:58.637 INFO:teuthology.orchestra.run.smithi132.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1979.gb117a47 4.0 MB/s | 1.8 MB 00:00 2024-03-11T22:45:58.829 INFO:teuthology.orchestra.run.smithi132.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.2 MB/s | 863 kB 00:00 2024-03-11T22:45:58.938 INFO:teuthology.orchestra.run.smithi132.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 65 kB/s | 19 kB 00:00 2024-03-11T22:45:59.288 INFO:teuthology.orchestra.run.smithi132.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 413 kB/s | 144 kB 00:00 2024-03-11T22:45:59.314 INFO:teuthology.orchestra.run.smithi132.stdout:(9/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 755 kB/s | 548 kB 00:00 2024-03-11T22:45:59.414 INFO:teuthology.orchestra.run.smithi132.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 245 kB/s | 31 kB 00:00 2024-03-11T22:45:59.439 INFO:teuthology.orchestra.run.smithi132.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.5 MB/s | 188 kB 00:00 2024-03-11T22:45:59.656 INFO:teuthology.orchestra.run.smithi132.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.5 MB/s | 2.0 MB 00:00 2024-03-11T22:45:59.660 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:45:59.660 INFO:teuthology.orchestra.run.smithi132.stdout:Total 3.7 MB/s | 6.9 MB 00:01 2024-03-11T22:45:59.680 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-11T22:45:59.700 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-11T22:45:59.700 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-11T22:45:59.875 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-11T22:45:59.876 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-11T22:46:00.184 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1979.gb117a 12/12 2024-03-11T22:46:00.184 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1979.gb117a 1/12 2024-03-11T22:46:00.184 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-11T22:46:00.184 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-11T22:46:00.184 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-11T22:46:00.184 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-11T22:46:00.184 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-11T22:46:00.184 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-11T22:46:00.184 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-11T22:46:00.185 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-11T22:46:00.185 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-11T22:46:00.185 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-11T22:46:00.396 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-11T22:46:00.465 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-11T22:46:00.508 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-11T22:46:00.554 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-11T22:46:00.585 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-11T22:46:00.604 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-11T22:46:00.604 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:00.604 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-03-11T22:46:00.604 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1979.gb117a47a.el9.noarch 2024-03-11T22:46:00.604 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-11T22:46:00.604 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-11T22:46:00.604 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-11T22:46:00.604 INFO:teuthology.orchestra.run.smithi078.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-11T22:46:00.604 INFO:teuthology.orchestra.run.smithi078.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-11T22:46:00.604 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-11T22:46:00.605 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-11T22:46:00.605 INFO:teuthology.orchestra.run.smithi078.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-11T22:46:00.605 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-11T22:46:00.605 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-11T22:46:00.605 INFO:teuthology.orchestra.run.smithi078.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-11T22:46:00.605 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:00.605 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-03-11T22:46:00.686 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-11T22:46:00.810 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-11T22:46:00.921 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-11T22:46:00.973 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-11T22:46:00.986 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-mgr-rook 2024-03-11T22:46:01.002 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-11T22:46:01.035 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1979.gb117a47a.el9 10/12 2024-03-11T22:46:01.278 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1979.gb117a47a.el 11/12 2024-03-11T22:46:01.317 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1979.gb117a47a.el9.noa 12/12 2024-03-11T22:46:01.562 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:19 ago on Mon 11 Mar 2024 10:44:42 PM UTC. 2024-03-11T22:46:01.712 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-03-11T22:46:01.714 INFO:teuthology.orchestra.run.smithi078.stdout:======================================================================================== 2024-03-11T22:46:01.714 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2024-03-11T22:46:01.714 INFO:teuthology.orchestra.run.smithi078.stdout:======================================================================================== 2024-03-11T22:46:01.714 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-03-11T22:46:01.714 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-rook noarch 2:19.0.0-1979.gb117a47a.el9 ceph-noarch 49 k 2024-03-11T22:46:01.714 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-03-11T22:46:01.714 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-11T22:46:01.714 INFO:teuthology.orchestra.run.smithi078.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-11T22:46:01.714 INFO:teuthology.orchestra.run.smithi078.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-11T22:46:01.714 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-11T22:46:01.715 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-11T22:46:01.715 INFO:teuthology.orchestra.run.smithi078.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-11T22:46:01.715 INFO:teuthology.orchestra.run.smithi078.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-11T22:46:01.715 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-11T22:46:01.715 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-11T22:46:01.715 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-11T22:46:01.715 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-11T22:46:01.715 INFO:teuthology.orchestra.run.smithi078.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-11T22:46:01.715 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:01.715 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-03-11T22:46:01.715 INFO:teuthology.orchestra.run.smithi078.stdout:======================================================================================== 2024-03-11T22:46:01.715 INFO:teuthology.orchestra.run.smithi078.stdout:Install 13 Packages 2024-03-11T22:46:01.715 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:01.716 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 2.0 M 2024-03-11T22:46:01.716 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 26 M 2024-03-11T22:46:01.716 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-03-11T22:46:01.930 INFO:teuthology.orchestra.run.smithi078.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 177 kB/s | 24 kB 00:00 2024-03-11T22:46:01.956 INFO:teuthology.orchestra.run.smithi078.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 111 kB/s | 18 kB 00:00 2024-03-11T22:46:02.030 INFO:teuthology.orchestra.run.smithi078.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-11T22:46:02.056 INFO:teuthology.orchestra.run.smithi078.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.7 MB/s | 174 kB 00:00 2024-03-11T22:46:02.081 INFO:teuthology.orchestra.run.smithi078.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-11T22:46:02.102 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1979.gb117a47a.el9.noa 12/12 2024-03-11T22:46:02.102 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1979.gb117a47a.el 1/12 2024-03-11T22:46:02.102 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1979.gb117a47a.el9.noa 2/12 2024-03-11T22:46:02.102 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1979.gb117a47a.el9 3/12 2024-03-11T22:46:02.103 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-11T22:46:02.103 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-11T22:46:02.103 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-11T22:46:02.103 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-11T22:46:02.103 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-11T22:46:02.103 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-11T22:46:02.103 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-11T22:46:02.103 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-11T22:46:02.106 INFO:teuthology.orchestra.run.smithi078.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2024-03-11T22:46:02.240 INFO:teuthology.orchestra.run.smithi078.stdout:(7/13): ceph-mgr-rook-19.0.0-1979.gb117a47a.el9 110 kB/s | 49 kB 00:00 2024-03-11T22:46:02.332 INFO:teuthology.orchestra.run.smithi078.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 128 kB/s | 32 kB 00:00 2024-03-11T22:46:02.357 INFO:teuthology.orchestra.run.smithi078.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 60 kB/s | 15 kB 00:00 2024-03-11T22:46:02.449 INFO:teuthology.orchestra.run.smithi078.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 654 kB/s | 59 kB 00:00 2024-03-11T22:46:02.462 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-11T22:46:02.463 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:02.463 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-11T22:46:02.463 INFO:teuthology.orchestra.run.smithi132.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-11T22:46:02.463 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-grafana-dashboards-2:19.0.0-1979.gb117a47a.el9.noarch 2024-03-11T22:46:02.463 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-dashboard-2:19.0.0-1979.gb117a47a.el9.noarch 2024-03-11T22:46:02.463 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-prometheus-alerts-2:19.0.0-1979.gb117a47a.el9.noarch 2024-03-11T22:46:02.463 INFO:teuthology.orchestra.run.smithi132.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-11T22:46:02.463 INFO:teuthology.orchestra.run.smithi132.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-11T22:46:02.463 INFO:teuthology.orchestra.run.smithi132.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-11T22:46:02.463 INFO:teuthology.orchestra.run.smithi132.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-11T22:46:02.463 INFO:teuthology.orchestra.run.smithi132.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-11T22:46:02.463 INFO:teuthology.orchestra.run.smithi132.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-11T22:46:02.463 INFO:teuthology.orchestra.run.smithi132.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-11T22:46:02.464 INFO:teuthology.orchestra.run.smithi132.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-11T22:46:02.464 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:02.464 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-11T22:46:02.532 INFO:teuthology.orchestra.run.smithi078.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.1 MB/s | 90 kB 00:00 2024-03-11T22:46:02.607 INFO:teuthology.orchestra.run.smithi078.stdout:(12/13): python3-google-auth-2.28.0-1.el9.noarc 551 kB/s | 203 kB 00:00 2024-03-11T22:46:02.690 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-11T22:46:02.833 INFO:teuthology.orchestra.run.smithi078.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.0 MB/s | 1.0 MB 00:00 2024-03-11T22:46:02.837 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:46:02.837 INFO:teuthology.orchestra.run.smithi078.stdout:Total 1.8 MB/s | 2.0 MB 00:01 2024-03-11T22:46:02.837 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-03-11T22:46:02.850 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-03-11T22:46:02.850 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-03-11T22:46:02.950 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-03-11T22:46:02.951 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-03-11T22:46:03.250 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:10 ago on Mon 11 Mar 2024 10:44:53 PM UTC. 2024-03-11T22:46:03.401 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-11T22:46:03.402 INFO:teuthology.orchestra.run.smithi132.stdout:============================================================================================ 2024-03-11T22:46:03.402 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-03-11T22:46:03.402 INFO:teuthology.orchestra.run.smithi132.stdout:============================================================================================ 2024-03-11T22:46:03.402 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-11T22:46:03.402 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1979.gb117a47a.el9 ceph-noarch 7.4 M 2024-03-11T22:46:03.403 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-03-11T22:46:03.403 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-11T22:46:03.403 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-11T22:46:03.403 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-11T22:46:03.403 INFO:teuthology.orchestra.run.smithi132.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-11T22:46:03.403 INFO:teuthology.orchestra.run.smithi132.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-11T22:46:03.403 INFO:teuthology.orchestra.run.smithi132.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-11T22:46:03.403 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-11T22:46:03.403 INFO:teuthology.orchestra.run.smithi132.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-11T22:46:03.403 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-11T22:46:03.403 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-11T22:46:03.403 INFO:teuthology.orchestra.run.smithi132.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-11T22:46:03.403 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:03.403 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-11T22:46:03.404 INFO:teuthology.orchestra.run.smithi132.stdout:============================================================================================ 2024-03-11T22:46:03.404 INFO:teuthology.orchestra.run.smithi132.stdout:Install 12 Packages 2024-03-11T22:46:03.404 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:03.404 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 37 M 2024-03-11T22:46:03.404 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 209 M 2024-03-11T22:46:03.404 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-11T22:46:03.498 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-03-11T22:46:03.672 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-11T22:46:03.734 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-11T22:46:03.798 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-11T22:46:03.835 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-11T22:46:03.876 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-11T22:46:03.938 INFO:teuthology.orchestra.run.smithi132.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 232 kB/s | 32 kB 00:00 2024-03-11T22:46:03.970 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-11T22:46:03.979 INFO:teuthology.orchestra.run.smithi132.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 414 kB/s | 17 kB 00:00 2024-03-11T22:46:04.022 INFO:teuthology.orchestra.run.smithi132.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 818 kB/s | 34 kB 00:00 2024-03-11T22:46:04.088 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 7/13 2024-03-11T22:46:04.145 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-11T22:46:04.197 INFO:teuthology.orchestra.run.smithi132.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2024-03-11T22:46:04.297 INFO:teuthology.orchestra.run.smithi132.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 48 MB/s | 4.8 MB 00:00 2024-03-11T22:46:04.356 INFO:teuthology.orchestra.run.smithi132.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.4 MB/s | 3.0 MB 00:00 2024-03-11T22:46:04.381 INFO:teuthology.orchestra.run.smithi132.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-03-11T22:46:04.606 INFO:teuthology.orchestra.run.smithi132.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 63 MB/s | 16 MB 00:00 2024-03-11T22:46:04.749 INFO:teuthology.orchestra.run.smithi132.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-19 7.8 MB/s | 7.4 MB 00:00 2024-03-11T22:46:04.782 INFO:teuthology.orchestra.run.smithi132.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.0 MB/s | 803 kB 00:00 2024-03-11T22:46:04.824 INFO:teuthology.orchestra.run.smithi132.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 901 kB/s | 195 kB 00:00 2024-03-11T22:46:04.875 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-11T22:46:04.917 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-11T22:46:04.950 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-11T22:46:04.999 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-11T22:46:05.020 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-rook-2:19.0.0-1979.gb117a47a.el9.noarch 13/13 2024-03-11T22:46:05.732 INFO:teuthology.orchestra.run.smithi132.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 249 kB/s | 245 kB 00:00 2024-03-11T22:46:05.735 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:46:05.736 INFO:teuthology.orchestra.run.smithi132.stdout:Total 16 MB/s | 37 MB 00:02 2024-03-11T22:46:05.778 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-11T22:46:05.803 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-11T22:46:05.804 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-11T22:46:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1979.gb117a47a.el9.noarch 13/13 2024-03-11T22:46:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1979.gb117a47a.el9.noarch 1/13 2024-03-11T22:46:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-11T22:46:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-11T22:46:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-11T22:46:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-11T22:46:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-11T22:46:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-11T22:46:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-11T22:46:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-11T22:46:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 10/13 2024-03-11T22:46:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-11T22:46:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-11T22:46:06.177 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-11T22:46:06.177 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:06.177 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-03-11T22:46:06.177 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-rook-2:19.0.0-1979.gb117a47a.el9.noarch 2024-03-11T22:46:06.177 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-11T22:46:06.177 INFO:teuthology.orchestra.run.smithi078.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-11T22:46:06.177 INFO:teuthology.orchestra.run.smithi078.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-11T22:46:06.177 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-11T22:46:06.177 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-11T22:46:06.178 INFO:teuthology.orchestra.run.smithi078.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-11T22:46:06.178 INFO:teuthology.orchestra.run.smithi078.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-11T22:46:06.178 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-11T22:46:06.178 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-11T22:46:06.178 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-11T22:46:06.178 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-11T22:46:06.178 INFO:teuthology.orchestra.run.smithi078.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-11T22:46:06.178 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:06.178 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-03-11T22:46:06.179 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-11T22:46:06.179 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-11T22:46:06.413 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-mgr-cephadm 2024-03-11T22:46:06.884 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-11T22:46:06.955 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-11T22:46:06.973 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:24 ago on Mon 11 Mar 2024 10:44:42 PM UTC. 2024-03-11T22:46:07.004 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-11T22:46:07.113 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-11T22:46:07.122 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-03-11T22:46:07.123 INFO:teuthology.orchestra.run.smithi078.stdout:======================================================================================== 2024-03-11T22:46:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2024-03-11T22:46:07.123 INFO:teuthology.orchestra.run.smithi078.stdout:======================================================================================== 2024-03-11T22:46:07.123 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-03-11T22:46:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1979.gb117a47a.el9 ceph-noarch 139 k 2024-03-11T22:46:07.123 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-03-11T22:46:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-11T22:46:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-11T22:46:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-11T22:46:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-11T22:46:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-11T22:46:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:07.124 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-03-11T22:46:07.124 INFO:teuthology.orchestra.run.smithi078.stdout:======================================================================================== 2024-03-11T22:46:07.124 INFO:teuthology.orchestra.run.smithi078.stdout:Install 6 Packages 2024-03-11T22:46:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:07.125 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 6.8 M 2024-03-11T22:46:07.125 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 33 M 2024-03-11T22:46:07.125 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-03-11T22:46:07.157 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-11T22:46:07.333 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-11T22:46:07.364 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-11T22:46:07.440 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-11T22:46:07.843 INFO:teuthology.orchestra.run.smithi078.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-11T22:46:07.893 INFO:teuthology.orchestra.run.smithi078.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1979.gb117a47a.e 422 kB/s | 139 kB 00:00 2024-03-11T22:46:08.094 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-11T22:46:08.201 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-11T22:46:08.235 INFO:teuthology.orchestra.run.smithi078.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 169 kB/s | 58 kB 00:00 2024-03-11T22:46:08.419 INFO:teuthology.orchestra.run.smithi078.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 278 kB/s | 51 kB 00:00 2024-03-11T22:46:08.486 INFO:teuthology.orchestra.run.smithi078.stdout:(5/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 884 kB/s | 548 kB 00:00 2024-03-11T22:46:08.636 INFO:teuthology.orchestra.run.smithi078.stdout:(6/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 231 kB/s | 248 kB 00:01 2024-03-11T22:46:08.638 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:46:08.638 INFO:teuthology.orchestra.run.smithi078.stdout:Total 4.5 MB/s | 6.8 MB 00:01 2024-03-11T22:46:08.652 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-03-11T22:46:08.664 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-03-11T22:46:08.664 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-03-11T22:46:08.760 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-03-11T22:46:08.760 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-03-11T22:46:09.061 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-03-11T22:46:09.228 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-11T22:46:09.271 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-03-11T22:46:09.440 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-11T22:46:09.796 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-11T22:46:09.890 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-11T22:46:09.951 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-11T22:46:10.008 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1979.gb117a47a.el9.noarch 6/6 2024-03-11T22:46:10.455 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-11T22:46:10.480 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1979.gb117a 12/12 2024-03-11T22:46:10.792 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1979.gb117a47a.el9.noarch 6/6 2024-03-11T22:46:10.792 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1979.gb117a47a.el9.noarch 1/6 2024-03-11T22:46:10.792 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-11T22:46:10.792 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-11T22:46:10.792 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-03-11T22:46:10.792 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-11T22:46:11.100 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-11T22:46:11.100 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:11.100 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-03-11T22:46:11.100 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-cephadm-2:19.0.0-1979.gb117a47a.el9.noarch 2024-03-11T22:46:11.100 INFO:teuthology.orchestra.run.smithi078.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-11T22:46:11.100 INFO:teuthology.orchestra.run.smithi078.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-11T22:46:11.100 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-11T22:46:11.100 INFO:teuthology.orchestra.run.smithi078.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-11T22:46:11.100 INFO:teuthology.orchestra.run.smithi078.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-11T22:46:11.100 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:11.101 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-03-11T22:46:11.378 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-fuse 2024-03-11T22:46:11.937 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:29 ago on Mon 11 Mar 2024 10:44:42 PM UTC. 2024-03-11T22:46:12.089 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-03-11T22:46:12.090 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-11T22:46:12.090 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2024-03-11T22:46:12.090 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-11T22:46:12.090 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-03-11T22:46:12.090 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-fuse x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 888 k 2024-03-11T22:46:12.090 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-03-11T22:46:12.090 INFO:teuthology.orchestra.run.smithi078.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-11T22:46:12.090 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:12.090 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-03-11T22:46:12.090 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-11T22:46:12.091 INFO:teuthology.orchestra.run.smithi078.stdout:Install 2 Packages 2024-03-11T22:46:12.091 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:12.091 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 967 k 2024-03-11T22:46:12.091 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 2.8 M 2024-03-11T22:46:12.091 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-03-11T22:46:12.220 INFO:teuthology.orchestra.run.smithi078.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 608 kB/s | 79 kB 00:00 2024-03-11T22:46:12.871 INFO:teuthology.orchestra.run.smithi078.stdout:(2/2): ceph-fuse-19.0.0-1979.gb117a47a.el9.x86_ 1.1 MB/s | 888 kB 00:00 2024-03-11T22:46:12.872 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:46:12.872 INFO:teuthology.orchestra.run.smithi078.stdout:Total 1.2 MB/s | 967 kB 00:00 2024-03-11T22:46:12.873 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-03-11T22:46:12.895 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-03-11T22:46:12.895 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-03-11T22:46:12.980 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-03-11T22:46:12.980 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-03-11T22:46:13.164 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-03-11T22:46:13.172 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1979.gb117a 12/12 2024-03-11T22:46:13.172 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1979.gb117a 1/12 2024-03-11T22:46:13.172 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-11T22:46:13.172 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-11T22:46:13.172 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-11T22:46:13.172 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-11T22:46:13.172 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-11T22:46:13.172 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-11T22:46:13.172 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-11T22:46:13.172 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-11T22:46:13.172 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-11T22:46:13.173 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-11T22:46:13.307 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-11T22:46:13.370 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-fuse-2:19.0.0-1979.gb117a47a.el9.x86_64 2/2 2024-03-11T22:46:13.673 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-11T22:46:13.674 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:13.674 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-11T22:46:13.674 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1979.gb117a47a.el9.noarch 2024-03-11T22:46:13.674 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-11T22:46:13.674 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-11T22:46:13.674 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-11T22:46:13.674 INFO:teuthology.orchestra.run.smithi132.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-11T22:46:13.674 INFO:teuthology.orchestra.run.smithi132.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-11T22:46:13.674 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-11T22:46:13.674 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-11T22:46:13.674 INFO:teuthology.orchestra.run.smithi132.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-11T22:46:13.674 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-11T22:46:13.675 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-11T22:46:13.675 INFO:teuthology.orchestra.run.smithi132.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-11T22:46:13.675 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:13.675 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-11T22:46:14.049 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr-rook 2024-03-11T22:46:14.077 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1979.gb117a47a.el9.x86_64 2/2 2024-03-11T22:46:14.078 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-fuse-2:19.0.0-1979.gb117a47a.el9.x86_64 1/2 2024-03-11T22:46:14.384 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-11T22:46:14.384 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:14.384 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-03-11T22:46:14.384 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-fuse-2:19.0.0-1979.gb117a47a.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-11T22:46:14.384 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:14.384 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-03-11T22:46:14.627 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:21 ago on Mon 11 Mar 2024 10:44:53 PM UTC. 2024-03-11T22:46:14.700 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-volume 2024-03-11T22:46:14.778 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-11T22:46:14.779 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================== 2024-03-11T22:46:14.780 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-03-11T22:46:14.780 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================== 2024-03-11T22:46:14.780 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-11T22:46:14.780 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-rook noarch 2:19.0.0-1979.gb117a47a.el9 ceph-noarch 49 k 2024-03-11T22:46:14.780 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-03-11T22:46:14.780 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-11T22:46:14.780 INFO:teuthology.orchestra.run.smithi132.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-11T22:46:14.780 INFO:teuthology.orchestra.run.smithi132.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-11T22:46:14.780 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-11T22:46:14.780 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-11T22:46:14.780 INFO:teuthology.orchestra.run.smithi132.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-11T22:46:14.780 INFO:teuthology.orchestra.run.smithi132.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-11T22:46:14.780 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-11T22:46:14.780 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-11T22:46:14.781 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-11T22:46:14.781 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-11T22:46:14.781 INFO:teuthology.orchestra.run.smithi132.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-11T22:46:14.781 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:14.781 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-11T22:46:14.781 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================== 2024-03-11T22:46:14.781 INFO:teuthology.orchestra.run.smithi132.stdout:Install 13 Packages 2024-03-11T22:46:14.781 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:14.782 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 2.0 M 2024-03-11T22:46:14.782 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 26 M 2024-03-11T22:46:14.782 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-11T22:46:15.033 INFO:teuthology.orchestra.run.smithi132.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 225 kB/s | 24 kB 00:00 2024-03-11T22:46:15.083 INFO:teuthology.orchestra.run.smithi132.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 114 kB/s | 18 kB 00:00 2024-03-11T22:46:15.142 INFO:teuthology.orchestra.run.smithi132.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 174 kB 00:00 2024-03-11T22:46:15.167 INFO:teuthology.orchestra.run.smithi132.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-11T22:46:15.192 INFO:teuthology.orchestra.run.smithi132.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-11T22:46:15.217 INFO:teuthology.orchestra.run.smithi132.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 873 kB/s | 43 kB 00:00 2024-03-11T22:46:15.277 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:33 ago on Mon 11 Mar 2024 10:44:42 PM UTC. 2024-03-11T22:46:15.429 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-03-11T22:46:15.429 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-11T22:46:15.429 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2024-03-11T22:46:15.429 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-11T22:46:15.429 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-03-11T22:46:15.429 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-volume noarch 2:19.0.0-1979.gb117a47a.el9 ceph-noarch 264 k 2024-03-11T22:46:15.430 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-03-11T22:46:15.430 INFO:teuthology.orchestra.run.smithi078.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-11T22:46:15.430 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:15.430 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-03-11T22:46:15.430 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-11T22:46:15.430 INFO:teuthology.orchestra.run.smithi078.stdout:Install 2 Packages 2024-03-11T22:46:15.430 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:15.431 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 332 k 2024-03-11T22:46:15.431 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 1.5 M 2024-03-11T22:46:15.431 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-03-11T22:46:15.451 INFO:teuthology.orchestra.run.smithi132.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 124 kB/s | 32 kB 00:00 2024-03-11T22:46:15.476 INFO:teuthology.orchestra.run.smithi132.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 58 kB/s | 15 kB 00:00 2024-03-11T22:46:15.559 INFO:teuthology.orchestra.run.smithi078.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 538 kB/s | 69 kB 00:00 2024-03-11T22:46:15.560 INFO:teuthology.orchestra.run.smithi132.stdout:(9/13): ceph-mgr-rook-19.0.0-1979.gb117a47a.el9 77 kB/s | 49 kB 00:00 2024-03-11T22:46:15.635 INFO:teuthology.orchestra.run.smithi132.stdout:(10/13): python3-google-auth-2.28.0-1.el9.noarc 1.1 MB/s | 203 kB 00:00 2024-03-11T22:46:15.727 INFO:teuthology.orchestra.run.smithi132.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 354 kB/s | 59 kB 00:00 2024-03-11T22:46:15.761 INFO:teuthology.orchestra.run.smithi132.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 3.6 MB/s | 1.0 MB 00:00 2024-03-11T22:46:15.786 INFO:teuthology.orchestra.run.smithi132.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 597 kB/s | 90 kB 00:00 2024-03-11T22:46:15.789 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:46:15.789 INFO:teuthology.orchestra.run.smithi132.stdout:Total 2.0 MB/s | 2.0 MB 00:01 2024-03-11T22:46:15.789 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-11T22:46:15.802 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-11T22:46:15.803 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-11T22:46:15.903 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-11T22:46:15.903 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-11T22:46:16.109 INFO:teuthology.orchestra.run.smithi078.stdout:(2/2): ceph-volume-19.0.0-1979.gb117a47a.el9.no 389 kB/s | 264 kB 00:00 2024-03-11T22:46:16.110 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:46:16.111 INFO:teuthology.orchestra.run.smithi078.stdout:Total 488 kB/s | 332 kB 00:00 2024-03-11T22:46:16.111 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-03-11T22:46:16.120 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-03-11T22:46:16.120 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-03-11T22:46:16.162 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-03-11T22:46:16.162 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-03-11T22:46:16.312 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-03-11T22:46:16.445 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-11T22:46:16.490 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-11T22:46:16.520 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-volume-2:19.0.0-1979.gb117a47a.el9.noarch 2/2 2024-03-11T22:46:16.535 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-volume-2:19.0.0-1979.gb117a47a.el9.noarch 2/2 2024-03-11T22:46:16.535 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:46:16.536 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-11T22:46:16.536 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:16.615 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-11T22:46:16.678 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-11T22:46:16.733 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-11T22:46:16.771 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-11T22:46:16.813 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-11T22:46:16.905 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-11T22:46:17.023 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 7/13 2024-03-11T22:46:17.073 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-11T22:46:17.283 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-volume-2:19.0.0-1979.gb117a47a.el9.noarch 1/2 2024-03-11T22:46:17.626 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-11T22:46:17.627 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:17.627 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-03-11T22:46:17.627 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-volume-2:19.0.0-1979.gb117a47a.el9.noarch 2024-03-11T22:46:17.627 INFO:teuthology.orchestra.run.smithi078.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-11T22:46:17.627 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:17.627 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-03-11T22:46:17.789 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-11T22:46:17.845 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-11T22:46:17.879 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-11T22:46:17.926 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-11T22:46:17.949 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-rook-2:19.0.0-1979.gb117a47a.el9.noarch 13/13 2024-03-11T22:46:17.958 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install librados-devel 2024-03-11T22:46:18.527 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:36 ago on Mon 11 Mar 2024 10:44:42 PM UTC. 2024-03-11T22:46:18.679 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-03-11T22:46:18.679 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-11T22:46:18.679 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repo Size 2024-03-11T22:46:18.679 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-11T22:46:18.680 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-03-11T22:46:18.680 INFO:teuthology.orchestra.run.smithi078.stdout: librados-devel x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 127 k 2024-03-11T22:46:18.680 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:18.680 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-03-11T22:46:18.680 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-11T22:46:18.680 INFO:teuthology.orchestra.run.smithi078.stdout:Install 1 Package 2024-03-11T22:46:18.680 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:18.680 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 127 k 2024-03-11T22:46:18.681 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 456 k 2024-03-11T22:46:18.681 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-03-11T22:46:18.769 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1979.gb117a47a.el9.noarch 13/13 2024-03-11T22:46:18.769 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1979.gb117a47a.el9.noarch 1/13 2024-03-11T22:46:18.769 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-11T22:46:18.769 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-11T22:46:18.769 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-11T22:46:18.769 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-11T22:46:18.770 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-11T22:46:18.770 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-11T22:46:18.770 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-11T22:46:18.770 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-11T22:46:18.770 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 10/13 2024-03-11T22:46:18.770 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-11T22:46:18.770 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-11T22:46:19.144 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-11T22:46:19.144 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:19.144 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-11T22:46:19.144 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-rook-2:19.0.0-1979.gb117a47a.el9.noarch 2024-03-11T22:46:19.144 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-11T22:46:19.144 INFO:teuthology.orchestra.run.smithi132.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-11T22:46:19.144 INFO:teuthology.orchestra.run.smithi132.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-11T22:46:19.144 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-11T22:46:19.144 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-11T22:46:19.145 INFO:teuthology.orchestra.run.smithi132.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-11T22:46:19.145 INFO:teuthology.orchestra.run.smithi132.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-11T22:46:19.145 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-11T22:46:19.145 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-11T22:46:19.145 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-11T22:46:19.145 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-11T22:46:19.145 INFO:teuthology.orchestra.run.smithi132.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-11T22:46:19.145 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:19.145 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-11T22:46:19.241 INFO:teuthology.orchestra.run.smithi078.stdout:librados-devel-19.0.0-1979.gb117a47a.el9.x86_64 226 kB/s | 127 kB 00:00 2024-03-11T22:46:19.242 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:46:19.242 INFO:teuthology.orchestra.run.smithi078.stdout:Total 225 kB/s | 127 kB 00:00 2024-03-11T22:46:19.242 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-03-11T22:46:19.248 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-03-11T22:46:19.248 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-03-11T22:46:19.310 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-03-11T22:46:19.310 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-03-11T22:46:19.401 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr-cephadm 2024-03-11T22:46:19.466 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-03-11T22:46:19.565 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librados-devel-2:19.0.0-1979.gb117a47a.el9.x86_64 1/1 2024-03-11T22:46:19.966 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:26 ago on Mon 11 Mar 2024 10:44:53 PM UTC. 2024-03-11T22:46:20.025 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librados-devel-2:19.0.0-1979.gb117a47a.el9.x86_64 1/1 2024-03-11T22:46:20.118 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-11T22:46:20.119 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================== 2024-03-11T22:46:20.119 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-03-11T22:46:20.119 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================== 2024-03-11T22:46:20.119 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-11T22:46:20.119 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1979.gb117a47a.el9 ceph-noarch 139 k 2024-03-11T22:46:20.119 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-03-11T22:46:20.119 INFO:teuthology.orchestra.run.smithi132.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-11T22:46:20.120 INFO:teuthology.orchestra.run.smithi132.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-11T22:46:20.120 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-11T22:46:20.120 INFO:teuthology.orchestra.run.smithi132.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-11T22:46:20.120 INFO:teuthology.orchestra.run.smithi132.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-11T22:46:20.120 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:20.120 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-11T22:46:20.120 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================== 2024-03-11T22:46:20.120 INFO:teuthology.orchestra.run.smithi132.stdout:Install 6 Packages 2024-03-11T22:46:20.120 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:20.121 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 6.8 M 2024-03-11T22:46:20.121 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 33 M 2024-03-11T22:46:20.121 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-11T22:46:20.331 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados-devel-2:19.0.0-1979.gb117a47a.el9.x86_64 1/1 2024-03-11T22:46:20.331 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:20.331 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-03-11T22:46:20.331 INFO:teuthology.orchestra.run.smithi078.stdout: librados-devel-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:46:20.331 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:20.331 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-03-11T22:46:20.593 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install libcephfs2 2024-03-11T22:46:20.797 INFO:teuthology.orchestra.run.smithi132.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-11T22:46:20.897 INFO:teuthology.orchestra.run.smithi132.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1979.gb117a47a.e 364 kB/s | 139 kB 00:00 2024-03-11T22:46:21.106 INFO:teuthology.orchestra.run.smithi132.stdout:(3/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 1.7 MB/s | 548 kB 00:00 2024-03-11T22:46:21.131 INFO:teuthology.orchestra.run.smithi132.stdout:(4/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 403 kB/s | 248 kB 00:00 2024-03-11T22:46:21.160 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:39 ago on Mon 11 Mar 2024 10:44:42 PM UTC. 2024-03-11T22:46:21.173 INFO:teuthology.orchestra.run.smithi132.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 764 kB/s | 51 kB 00:00 2024-03-11T22:46:21.259 INFO:teuthology.orchestra.run.smithi078.stdout:Package libcephfs2-2:19.0.0-1979.gb117a47a.el9.x86_64 is already installed. 2024-03-11T22:46:21.316 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-03-11T22:46:21.318 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2024-03-11T22:46:21.318 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-03-11T22:46:21.394 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install libcephfs-devel 2024-03-11T22:46:21.957 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:39 ago on Mon 11 Mar 2024 10:44:42 PM UTC. 2024-03-11T22:46:22.106 INFO:teuthology.orchestra.run.smithi132.stdout:(6/6): python3-natsort-7.1.1-5.el9.noarch.rpm 48 kB/s | 58 kB 00:01 2024-03-11T22:46:22.107 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:46:22.108 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-03-11T22:46:22.108 INFO:teuthology.orchestra.run.smithi132.stdout:Total 3.4 MB/s | 6.8 MB 00:01 2024-03-11T22:46:22.108 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-11T22:46:22.109 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repo Size 2024-03-11T22:46:22.109 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-11T22:46:22.109 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-03-11T22:46:22.109 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs-devel x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 31 k 2024-03-11T22:46:22.109 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:22.109 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-03-11T22:46:22.109 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-11T22:46:22.109 INFO:teuthology.orchestra.run.smithi078.stdout:Install 1 Package 2024-03-11T22:46:22.109 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:22.109 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 31 k 2024-03-11T22:46:22.109 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 140 k 2024-03-11T22:46:22.109 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-03-11T22:46:22.121 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-11T22:46:22.131 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-11T22:46:22.131 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-11T22:46:22.224 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-11T22:46:22.233 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-11T22:46:22.277 INFO:teuthology.orchestra.run.smithi078.stdout:libcephfs-devel-19.0.0-1979.gb117a47a.el9.x86_6 185 kB/s | 31 kB 00:00 2024-03-11T22:46:22.278 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:46:22.278 INFO:teuthology.orchestra.run.smithi078.stdout:Total 184 kB/s | 31 kB 00:00 2024-03-11T22:46:22.278 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-03-11T22:46:22.281 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-03-11T22:46:22.282 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-03-11T22:46:22.309 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-03-11T22:46:22.309 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-03-11T22:46:22.434 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-03-11T22:46:22.527 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-11T22:46:22.551 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephfs-devel-2:19.0.0-1979.gb117a47a.el9.x86_64 1/1 2024-03-11T22:46:22.689 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-11T22:46:22.734 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-03-11T22:46:22.930 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1979.gb117a47a.el9.x86_64 1/1 2024-03-11T22:46:23.199 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephfs-devel-2:19.0.0-1979.gb117a47a.el9.x86_64 1/1 2024-03-11T22:46:23.200 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:23.200 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-03-11T22:46:23.200 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs-devel-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:46:23.200 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:23.200 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-03-11T22:46:23.258 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-11T22:46:23.337 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-11T22:46:23.381 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install librados2 2024-03-11T22:46:23.398 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-11T22:46:23.422 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1979.gb117a47a.el9.noarch 6/6 2024-03-11T22:46:23.939 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:41 ago on Mon 11 Mar 2024 10:44:42 PM UTC. 2024-03-11T22:46:24.035 INFO:teuthology.orchestra.run.smithi078.stdout:Package librados2-2:19.0.0-1979.gb117a47a.el9.x86_64 is already installed. 2024-03-11T22:46:24.087 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-03-11T22:46:24.088 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2024-03-11T22:46:24.088 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-03-11T22:46:24.133 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install librbd1 2024-03-11T22:46:24.246 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1979.gb117a47a.el9.noarch 6/6 2024-03-11T22:46:24.247 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1979.gb117a47a.el9.noarch 1/6 2024-03-11T22:46:24.248 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-11T22:46:24.248 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-11T22:46:24.248 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-03-11T22:46:24.248 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-11T22:46:24.567 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-11T22:46:24.567 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:24.568 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-11T22:46:24.568 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-cephadm-2:19.0.0-1979.gb117a47a.el9.noarch 2024-03-11T22:46:24.568 INFO:teuthology.orchestra.run.smithi132.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-11T22:46:24.568 INFO:teuthology.orchestra.run.smithi132.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-11T22:46:24.568 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-11T22:46:24.568 INFO:teuthology.orchestra.run.smithi132.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-11T22:46:24.568 INFO:teuthology.orchestra.run.smithi132.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-11T22:46:24.568 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:24.568 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-11T22:46:24.693 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:42 ago on Mon 11 Mar 2024 10:44:42 PM UTC. 2024-03-11T22:46:24.789 INFO:teuthology.orchestra.run.smithi078.stdout:Package librbd1-2:19.0.0-1979.gb117a47a.el9.x86_64 is already installed. 2024-03-11T22:46:24.806 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-fuse 2024-03-11T22:46:24.840 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-03-11T22:46:24.841 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2024-03-11T22:46:24.841 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-03-11T22:46:24.894 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install python3-rados 2024-03-11T22:46:25.367 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:32 ago on Mon 11 Mar 2024 10:44:53 PM UTC. 2024-03-11T22:46:25.455 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:43 ago on Mon 11 Mar 2024 10:44:42 PM UTC. 2024-03-11T22:46:25.516 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-11T22:46:25.517 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-11T22:46:25.517 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-03-11T22:46:25.517 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-11T22:46:25.517 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-11T22:46:25.517 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-fuse x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 888 k 2024-03-11T22:46:25.517 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-03-11T22:46:25.517 INFO:teuthology.orchestra.run.smithi132.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-11T22:46:25.518 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:25.518 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-11T22:46:25.518 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-11T22:46:25.518 INFO:teuthology.orchestra.run.smithi132.stdout:Install 2 Packages 2024-03-11T22:46:25.518 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:25.519 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 967 k 2024-03-11T22:46:25.519 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 2.8 M 2024-03-11T22:46:25.519 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-11T22:46:25.552 INFO:teuthology.orchestra.run.smithi078.stdout:Package python3-rados-2:19.0.0-1979.gb117a47a.el9.x86_64 is already installed. 2024-03-11T22:46:25.604 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-03-11T22:46:25.605 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2024-03-11T22:46:25.605 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-03-11T22:46:25.653 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install python3-rgw 2024-03-11T22:46:25.710 INFO:teuthology.orchestra.run.smithi132.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 411 kB/s | 79 kB 00:00 2024-03-11T22:46:25.955 INFO:teuthology.orchestra.run.smithi132.stdout:(2/2): ceph-fuse-19.0.0-1979.gb117a47a.el9.x86_ 2.0 MB/s | 888 kB 00:00 2024-03-11T22:46:25.956 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:46:25.956 INFO:teuthology.orchestra.run.smithi132.stdout:Total 2.2 MB/s | 967 kB 00:00 2024-03-11T22:46:25.956 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-11T22:46:25.976 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-11T22:46:25.977 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-11T22:46:26.058 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-11T22:46:26.058 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-11T22:46:26.213 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:44 ago on Mon 11 Mar 2024 10:44:42 PM UTC. 2024-03-11T22:46:26.256 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-11T22:46:26.310 INFO:teuthology.orchestra.run.smithi078.stdout:Package python3-rgw-2:19.0.0-1979.gb117a47a.el9.x86_64 is already installed. 2024-03-11T22:46:26.361 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-03-11T22:46:26.363 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2024-03-11T22:46:26.363 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-03-11T22:46:26.386 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-11T22:46:26.409 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install python3-cephfs 2024-03-11T22:46:26.460 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-fuse-2:19.0.0-1979.gb117a47a.el9.x86_64 2/2 2024-03-11T22:46:26.967 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:44 ago on Mon 11 Mar 2024 10:44:42 PM UTC. 2024-03-11T22:46:27.065 INFO:teuthology.orchestra.run.smithi078.stdout:Package python3-cephfs-2:19.0.0-1979.gb117a47a.el9.x86_64 is already installed. 2024-03-11T22:46:27.116 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-03-11T22:46:27.117 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2024-03-11T22:46:27.117 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-03-11T22:46:27.125 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1979.gb117a47a.el9.x86_64 2/2 2024-03-11T22:46:27.125 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-fuse-2:19.0.0-1979.gb117a47a.el9.x86_64 1/2 2024-03-11T22:46:27.171 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install python3-rbd 2024-03-11T22:46:27.435 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-11T22:46:27.435 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:27.435 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-11T22:46:27.435 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-fuse-2:19.0.0-1979.gb117a47a.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-11T22:46:27.435 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:27.435 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-11T22:46:27.723 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-volume 2024-03-11T22:46:27.730 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:45 ago on Mon 11 Mar 2024 10:44:42 PM UTC. 2024-03-11T22:46:27.827 INFO:teuthology.orchestra.run.smithi078.stdout:Package python3-rbd-2:19.0.0-1979.gb117a47a.el9.x86_64 is already installed. 2024-03-11T22:46:27.878 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-03-11T22:46:27.879 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2024-03-11T22:46:27.879 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-03-11T22:46:27.927 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install rbd-fuse 2024-03-11T22:46:28.299 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:35 ago on Mon 11 Mar 2024 10:44:53 PM UTC. 2024-03-11T22:46:28.454 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-11T22:46:28.454 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-11T22:46:28.455 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-03-11T22:46:28.455 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-11T22:46:28.455 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-11T22:46:28.455 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-volume noarch 2:19.0.0-1979.gb117a47a.el9 ceph-noarch 264 k 2024-03-11T22:46:28.455 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-03-11T22:46:28.455 INFO:teuthology.orchestra.run.smithi132.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-11T22:46:28.455 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:28.455 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-11T22:46:28.455 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-11T22:46:28.455 INFO:teuthology.orchestra.run.smithi132.stdout:Install 2 Packages 2024-03-11T22:46:28.455 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:28.456 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 332 k 2024-03-11T22:46:28.456 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 1.5 M 2024-03-11T22:46:28.456 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-11T22:46:28.502 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:46 ago on Mon 11 Mar 2024 10:44:42 PM UTC. 2024-03-11T22:46:28.598 INFO:teuthology.orchestra.run.smithi132.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 484 kB/s | 69 kB 00:00 2024-03-11T22:46:28.661 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-03-11T22:46:28.661 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-11T22:46:28.661 INFO:teuthology.orchestra.run.smithi078.stdout: Package Architecture Version Repository Size 2024-03-11T22:46:28.662 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-11T22:46:28.662 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-03-11T22:46:28.662 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-fuse x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 85 k 2024-03-11T22:46:28.662 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:28.662 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-03-11T22:46:28.662 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-11T22:46:28.662 INFO:teuthology.orchestra.run.smithi078.stdout:Install 1 Package 2024-03-11T22:46:28.662 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:28.662 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 85 k 2024-03-11T22:46:28.662 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 231 k 2024-03-11T22:46:28.662 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-03-11T22:46:28.749 INFO:teuthology.orchestra.run.smithi132.stdout:(2/2): ceph-volume-19.0.0-1979.gb117a47a.el9.no 903 kB/s | 264 kB 00:00 2024-03-11T22:46:28.750 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:46:28.750 INFO:teuthology.orchestra.run.smithi132.stdout:Total 1.1 MB/s | 332 kB 00:00 2024-03-11T22:46:28.750 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-11T22:46:28.758 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-11T22:46:28.759 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-11T22:46:28.804 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-11T22:46:28.804 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-11T22:46:28.952 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-11T22:46:28.998 INFO:teuthology.orchestra.run.smithi078.stdout:rbd-fuse-19.0.0-1979.gb117a47a.el9.x86_64.rpm 254 kB/s | 85 kB 00:00 2024-03-11T22:46:28.999 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:46:28.999 INFO:teuthology.orchestra.run.smithi078.stdout:Total 253 kB/s | 85 kB 00:00 2024-03-11T22:46:28.999 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-03-11T22:46:29.006 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-03-11T22:46:29.006 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-03-11T22:46:29.063 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-03-11T22:46:29.064 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-03-11T22:46:29.121 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-11T22:46:29.168 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-volume-2:19.0.0-1979.gb117a47a.el9.noarch 2/2 2024-03-11T22:46:29.184 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-volume-2:19.0.0-1979.gb117a47a.el9.noarch 2/2 2024-03-11T22:46:29.184 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:46:29.184 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-11T22:46:29.184 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:29.207 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-03-11T22:46:29.315 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-fuse-2:19.0.0-1979.gb117a47a.el9.x86_64 1/1 2024-03-11T22:46:29.727 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1979.gb117a47a.el9.x86_64 1/1 2024-03-11T22:46:29.899 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-volume-2:19.0.0-1979.gb117a47a.el9.noarch 1/2 2024-03-11T22:46:30.056 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-fuse-2:19.0.0-1979.gb117a47a.el9.x86_64 1/1 2024-03-11T22:46:30.056 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:30.056 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-03-11T22:46:30.056 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-fuse-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:46:30.056 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:30.057 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-03-11T22:46:30.252 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-11T22:46:30.252 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:30.252 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-11T22:46:30.252 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-volume-2:19.0.0-1979.gb117a47a.el9.noarch 2024-03-11T22:46:30.252 INFO:teuthology.orchestra.run.smithi132.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-11T22:46:30.252 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:30.252 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-11T22:46:30.256 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install rbd-mirror 2024-03-11T22:46:30.577 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install librados-devel 2024-03-11T22:46:30.829 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:48 ago on Mon 11 Mar 2024 10:44:42 PM UTC. 2024-03-11T22:46:30.982 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-03-11T22:46:30.983 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-11T22:46:30.983 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repo Size 2024-03-11T22:46:30.983 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-11T22:46:30.983 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-03-11T22:46:30.983 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-mirror x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 3.1 M 2024-03-11T22:46:30.983 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:30.983 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-03-11T22:46:30.983 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-11T22:46:30.983 INFO:teuthology.orchestra.run.smithi078.stdout:Install 1 Package 2024-03-11T22:46:30.983 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:30.983 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 3.1 M 2024-03-11T22:46:30.983 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 13 M 2024-03-11T22:46:30.984 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-03-11T22:46:31.156 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:38 ago on Mon 11 Mar 2024 10:44:53 PM UTC. 2024-03-11T22:46:31.309 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-11T22:46:31.310 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-11T22:46:31.310 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-03-11T22:46:31.310 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-11T22:46:31.310 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-11T22:46:31.310 INFO:teuthology.orchestra.run.smithi132.stdout: librados-devel x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 127 k 2024-03-11T22:46:31.310 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:31.311 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-11T22:46:31.311 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-11T22:46:31.311 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-11T22:46:31.311 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:31.311 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 127 k 2024-03-11T22:46:31.311 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 456 k 2024-03-11T22:46:31.311 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-11T22:46:31.717 INFO:teuthology.orchestra.run.smithi078.stdout:rbd-mirror-19.0.0-1979.gb117a47a.el9.x86_64.rpm 4.2 MB/s | 3.1 MB 00:00 2024-03-11T22:46:31.718 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:46:31.718 INFO:teuthology.orchestra.run.smithi078.stdout:Total 4.2 MB/s | 3.1 MB 00:00 2024-03-11T22:46:31.718 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-03-11T22:46:31.727 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-03-11T22:46:31.728 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-03-11T22:46:31.800 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-03-11T22:46:31.800 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-03-11T22:46:32.227 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-03-11T22:46:32.285 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-mirror-2:19.0.0-1979.gb117a47a.el9.x86_64 1/1 2024-03-11T22:46:32.310 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1979.gb117a47a.el9.x86_64 1/1 2024-03-11T22:46:32.310 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:46:32.310 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-11T22:46:32.310 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T22:46:32.310 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T22:46:32.310 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:32.322 INFO:teuthology.orchestra.run.smithi132.stdout:librados-devel-19.0.0-1979.gb117a47a.el9.x86_64 125 kB/s | 127 kB 00:01 2024-03-11T22:46:32.323 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:46:32.323 INFO:teuthology.orchestra.run.smithi132.stdout:Total 125 kB/s | 127 kB 00:01 2024-03-11T22:46:32.323 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-11T22:46:32.330 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-11T22:46:32.330 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-11T22:46:32.397 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-11T22:46:32.397 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-11T22:46:32.542 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-11T22:46:32.646 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librados-devel-2:19.0.0-1979.gb117a47a.el9.x86_64 1/1 2024-03-11T22:46:33.081 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados-devel-2:19.0.0-1979.gb117a47a.el9.x86_64 1/1 2024-03-11T22:46:33.384 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados-devel-2:19.0.0-1979.gb117a47a.el9.x86_64 1/1 2024-03-11T22:46:33.384 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:33.385 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-11T22:46:33.385 INFO:teuthology.orchestra.run.smithi132.stdout: librados-devel-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:46:33.385 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:33.385 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-11T22:46:33.468 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-mirror-2:19.0.0-1979.gb117a47a.el9.x86_64 1/1 2024-03-11T22:46:33.468 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:33.468 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-03-11T22:46:33.469 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-mirror-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:46:33.469 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:33.469 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-03-11T22:46:33.596 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install libcephfs2 2024-03-11T22:46:33.705 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install rbd-nbd 2024-03-11T22:46:34.168 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:41 ago on Mon 11 Mar 2024 10:44:53 PM UTC. 2024-03-11T22:46:34.267 INFO:teuthology.orchestra.run.smithi132.stdout:Package libcephfs2-2:19.0.0-1979.gb117a47a.el9.x86_64 is already installed. 2024-03-11T22:46:34.279 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:52 ago on Mon 11 Mar 2024 10:44:42 PM UTC. 2024-03-11T22:46:34.323 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-11T22:46:34.324 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-03-11T22:46:34.324 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-11T22:46:34.388 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install libcephfs-devel 2024-03-11T22:46:34.427 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-03-11T22:46:34.428 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-11T22:46:34.428 INFO:teuthology.orchestra.run.smithi078.stdout: Package Architecture Version Repository Size 2024-03-11T22:46:34.428 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-11T22:46:34.428 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-03-11T22:46:34.428 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-nbd x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 170 k 2024-03-11T22:46:34.428 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:34.428 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-03-11T22:46:34.428 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-11T22:46:34.429 INFO:teuthology.orchestra.run.smithi078.stdout:Install 1 Package 2024-03-11T22:46:34.429 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:34.429 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 170 k 2024-03-11T22:46:34.429 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 490 k 2024-03-11T22:46:34.429 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-03-11T22:46:34.954 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:41 ago on Mon 11 Mar 2024 10:44:53 PM UTC. 2024-03-11T22:46:35.105 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-11T22:46:35.106 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-11T22:46:35.106 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-03-11T22:46:35.106 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-11T22:46:35.106 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-11T22:46:35.107 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs-devel x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 31 k 2024-03-11T22:46:35.107 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:35.107 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-11T22:46:35.107 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-11T22:46:35.107 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-11T22:46:35.107 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:35.107 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 31 k 2024-03-11T22:46:35.107 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 140 k 2024-03-11T22:46:35.107 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-11T22:46:35.204 INFO:teuthology.orchestra.run.smithi078.stdout:rbd-nbd-19.0.0-1979.gb117a47a.el9.x86_64.rpm 220 kB/s | 170 kB 00:00 2024-03-11T22:46:35.205 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:46:35.205 INFO:teuthology.orchestra.run.smithi078.stdout:Total 219 kB/s | 170 kB 00:00 2024-03-11T22:46:35.205 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-03-11T22:46:35.213 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-03-11T22:46:35.213 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-03-11T22:46:35.269 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-03-11T22:46:35.269 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-03-11T22:46:35.451 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-03-11T22:46:35.558 INFO:teuthology.orchestra.run.smithi132.stdout:libcephfs-devel-19.0.0-1979.gb117a47a.el9.x86_6 69 kB/s | 31 kB 00:00 2024-03-11T22:46:35.559 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:46:35.559 INFO:teuthology.orchestra.run.smithi132.stdout:Total 69 kB/s | 31 kB 00:00 2024-03-11T22:46:35.559 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-11T22:46:35.562 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-11T22:46:35.562 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-11T22:46:35.569 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-nbd-2:19.0.0-1979.gb117a47a.el9.x86_64 1/1 2024-03-11T22:46:35.601 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-11T22:46:35.602 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-11T22:46:35.731 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-11T22:46:35.848 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephfs-devel-2:19.0.0-1979.gb117a47a.el9.x86_64 1/1 2024-03-11T22:46:36.009 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1979.gb117a47a.el9.x86_64 1/1 2024-03-11T22:46:36.249 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1979.gb117a47a.el9.x86_64 1/1 2024-03-11T22:46:36.338 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-nbd-2:19.0.0-1979.gb117a47a.el9.x86_64 1/1 2024-03-11T22:46:36.338 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:36.338 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-03-11T22:46:36.338 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-nbd-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:46:36.338 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:36.338 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-03-11T22:46:36.526 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephfs-devel-2:19.0.0-1979.gb117a47a.el9.x86_64 1/1 2024-03-11T22:46:36.526 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:36.526 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-11T22:46:36.527 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs-devel-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:46:36.527 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:36.527 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-11T22:46:36.547 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install rbd-mirror 2024-03-11T22:46:36.684 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install librados2 2024-03-11T22:46:37.112 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:55 ago on Mon 11 Mar 2024 10:44:42 PM UTC. 2024-03-11T22:46:37.212 INFO:teuthology.orchestra.run.smithi078.stdout:Package rbd-mirror-2:19.0.0-1979.gb117a47a.el9.x86_64 is already installed. 2024-03-11T22:46:37.248 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:44 ago on Mon 11 Mar 2024 10:44:53 PM UTC. 2024-03-11T22:46:37.266 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-03-11T22:46:37.267 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2024-03-11T22:46:37.268 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-03-11T22:46:37.327 DEBUG:teuthology.parallel:result is None 2024-03-11T22:46:37.345 INFO:teuthology.orchestra.run.smithi132.stdout:Package librados2-2:19.0.0-1979.gb117a47a.el9.x86_64 is already installed. 2024-03-11T22:46:37.397 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-11T22:46:37.398 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-03-11T22:46:37.398 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-11T22:46:37.442 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install librbd1 2024-03-11T22:46:38.003 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:44 ago on Mon 11 Mar 2024 10:44:53 PM UTC. 2024-03-11T22:46:38.101 INFO:teuthology.orchestra.run.smithi132.stdout:Package librbd1-2:19.0.0-1979.gb117a47a.el9.x86_64 is already installed. 2024-03-11T22:46:38.152 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-11T22:46:38.153 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-03-11T22:46:38.153 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-11T22:46:38.200 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-rados 2024-03-11T22:46:38.763 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:45 ago on Mon 11 Mar 2024 10:44:53 PM UTC. 2024-03-11T22:46:38.860 INFO:teuthology.orchestra.run.smithi132.stdout:Package python3-rados-2:19.0.0-1979.gb117a47a.el9.x86_64 is already installed. 2024-03-11T22:46:38.912 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-11T22:46:38.913 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-03-11T22:46:38.913 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-11T22:46:38.955 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-rgw 2024-03-11T22:46:39.519 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:46 ago on Mon 11 Mar 2024 10:44:53 PM UTC. 2024-03-11T22:46:39.616 INFO:teuthology.orchestra.run.smithi132.stdout:Package python3-rgw-2:19.0.0-1979.gb117a47a.el9.x86_64 is already installed. 2024-03-11T22:46:39.667 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-11T22:46:39.668 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-03-11T22:46:39.668 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-11T22:46:39.717 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-cephfs 2024-03-11T22:46:40.288 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:47 ago on Mon 11 Mar 2024 10:44:53 PM UTC. 2024-03-11T22:46:40.396 INFO:teuthology.orchestra.run.smithi132.stdout:Package python3-cephfs-2:19.0.0-1979.gb117a47a.el9.x86_64 is already installed. 2024-03-11T22:46:40.451 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-11T22:46:40.452 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-03-11T22:46:40.452 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-11T22:46:40.503 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-rbd 2024-03-11T22:46:41.096 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:48 ago on Mon 11 Mar 2024 10:44:53 PM UTC. 2024-03-11T22:46:41.197 INFO:teuthology.orchestra.run.smithi132.stdout:Package python3-rbd-2:19.0.0-1979.gb117a47a.el9.x86_64 is already installed. 2024-03-11T22:46:41.252 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-11T22:46:41.253 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-03-11T22:46:41.253 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-11T22:46:41.297 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install rbd-fuse 2024-03-11T22:46:41.864 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:48 ago on Mon 11 Mar 2024 10:44:53 PM UTC. 2024-03-11T22:46:42.012 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-11T22:46:42.012 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-11T22:46:42.012 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2024-03-11T22:46:42.013 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-11T22:46:42.013 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-11T22:46:42.013 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-fuse x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 85 k 2024-03-11T22:46:42.013 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:42.013 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-11T22:46:42.013 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-11T22:46:42.013 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-11T22:46:42.013 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:42.013 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 85 k 2024-03-11T22:46:42.013 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 231 k 2024-03-11T22:46:42.013 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-11T22:46:42.445 INFO:teuthology.orchestra.run.smithi132.stdout:rbd-fuse-19.0.0-1979.gb117a47a.el9.x86_64.rpm 197 kB/s | 85 kB 00:00 2024-03-11T22:46:42.446 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:46:42.446 INFO:teuthology.orchestra.run.smithi132.stdout:Total 197 kB/s | 85 kB 00:00 2024-03-11T22:46:42.446 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-11T22:46:42.453 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-11T22:46:42.453 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-11T22:46:42.508 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-11T22:46:42.508 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-11T22:46:42.647 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-11T22:46:42.752 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-fuse-2:19.0.0-1979.gb117a47a.el9.x86_64 1/1 2024-03-11T22:46:43.171 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1979.gb117a47a.el9.x86_64 1/1 2024-03-11T22:46:43.458 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-fuse-2:19.0.0-1979.gb117a47a.el9.x86_64 1/1 2024-03-11T22:46:43.459 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:43.459 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-11T22:46:43.459 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-fuse-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:46:43.459 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:43.459 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-11T22:46:43.681 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install rbd-mirror 2024-03-11T22:46:44.254 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:51 ago on Mon 11 Mar 2024 10:44:53 PM UTC. 2024-03-11T22:46:44.408 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-11T22:46:44.409 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-11T22:46:44.409 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-03-11T22:46:44.409 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-11T22:46:44.409 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-11T22:46:44.409 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-mirror x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 3.1 M 2024-03-11T22:46:44.409 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:44.409 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-11T22:46:44.409 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-11T22:46:44.409 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-11T22:46:44.409 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:44.410 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 3.1 M 2024-03-11T22:46:44.410 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 13 M 2024-03-11T22:46:44.410 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-11T22:46:45.115 INFO:teuthology.orchestra.run.smithi132.stdout:rbd-mirror-19.0.0-1979.gb117a47a.el9.x86_64.rpm 4.4 MB/s | 3.1 MB 00:00 2024-03-11T22:46:45.116 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:46:45.116 INFO:teuthology.orchestra.run.smithi132.stdout:Total 4.4 MB/s | 3.1 MB 00:00 2024-03-11T22:46:45.116 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-11T22:46:45.126 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-11T22:46:45.126 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-11T22:46:45.198 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-11T22:46:45.199 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-11T22:46:45.624 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-11T22:46:45.691 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-mirror-2:19.0.0-1979.gb117a47a.el9.x86_64 1/1 2024-03-11T22:46:45.716 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1979.gb117a47a.el9.x86_64 1/1 2024-03-11T22:46:45.716 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:46:45.716 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-11T22:46:45.716 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T22:46:45.716 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T22:46:45.716 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:46.842 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-mirror-2:19.0.0-1979.gb117a47a.el9.x86_64 1/1 2024-03-11T22:46:46.842 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:46.842 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-11T22:46:46.842 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-mirror-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:46:46.842 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:46.842 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-11T22:46:47.073 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install rbd-nbd 2024-03-11T22:46:47.652 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:54 ago on Mon 11 Mar 2024 10:44:53 PM UTC. 2024-03-11T22:46:47.800 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-11T22:46:47.801 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-11T22:46:47.801 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2024-03-11T22:46:47.801 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-11T22:46:47.801 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-11T22:46:47.801 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-nbd x86_64 2:19.0.0-1979.gb117a47a.el9 ceph 170 k 2024-03-11T22:46:47.801 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:47.801 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-11T22:46:47.801 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-11T22:46:47.801 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-11T22:46:47.801 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:47.801 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 170 k 2024-03-11T22:46:47.801 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 490 k 2024-03-11T22:46:47.801 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-11T22:46:48.169 INFO:teuthology.orchestra.run.smithi132.stdout:rbd-nbd-19.0.0-1979.gb117a47a.el9.x86_64.rpm 464 kB/s | 170 kB 00:00 2024-03-11T22:46:48.170 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:46:48.170 INFO:teuthology.orchestra.run.smithi132.stdout:Total 462 kB/s | 170 kB 00:00 2024-03-11T22:46:48.170 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-11T22:46:48.177 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-11T22:46:48.177 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-11T22:46:48.233 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-11T22:46:48.233 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-11T22:46:48.383 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-11T22:46:48.484 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-nbd-2:19.0.0-1979.gb117a47a.el9.x86_64 1/1 2024-03-11T22:46:48.937 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1979.gb117a47a.el9.x86_64 1/1 2024-03-11T22:46:49.248 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-nbd-2:19.0.0-1979.gb117a47a.el9.x86_64 1/1 2024-03-11T22:46:49.248 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:49.248 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-11T22:46:49.248 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-nbd-2:19.0.0-1979.gb117a47a.el9.x86_64 2024-03-11T22:46:49.248 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:49.248 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-11T22:46:49.477 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install rbd-mirror 2024-03-11T22:46:50.049 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:57 ago on Mon 11 Mar 2024 10:44:53 PM UTC. 2024-03-11T22:46:50.153 INFO:teuthology.orchestra.run.smithi132.stdout:Package rbd-mirror-2:19.0.0-1979.gb117a47a.el9.x86_64 is already installed. 2024-03-11T22:46:50.206 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-11T22:46:50.207 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-03-11T22:46:50.207 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-11T22:46:50.265 DEBUG:teuthology.parallel:result is None 2024-03-11T22:46:50.265 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b117a47aa0ad81d207fdd9ba852068f2f262fea8 2024-03-11T22:46:50.485 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-11T22:46:50.509 INFO:teuthology.orchestra.run.smithi078.stdout:19.0.0-1979.gb117a47a.el9 2024-03-11T22:46:50.510 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1979.gb117a47a.el9 2024-03-11T22:46:50.510 INFO:teuthology.task.install:The correct ceph version 19.0.0-1979.gb117a47a is installed. 2024-03-11T22:46:50.512 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b117a47aa0ad81d207fdd9ba852068f2f262fea8 2024-03-11T22:46:50.747 DEBUG:teuthology.orchestra.run.smithi132:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-11T22:46:50.768 INFO:teuthology.orchestra.run.smithi132.stdout:19.0.0-1979.gb117a47a.el9 2024-03-11T22:46:50.769 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1979.gb117a47a.el9 2024-03-11T22:46:50.769 INFO:teuthology.task.install:The correct ceph version 19.0.0-1979.gb117a47a is installed. 2024-03-11T22:46:50.771 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-11T22:46:50.793 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-11T22:46:50.793 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-11T22:46:50.825 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:46:50.825 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-11T22:46:50.852 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-11T22:46:50.852 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-11T22:46:50.852 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/daemon-helper 2024-03-11T22:46:50.895 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-11T22:46:50.965 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:46:50.965 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/daemon-helper 2024-03-11T22:46:50.994 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-11T22:46:51.061 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-11T22:46:51.062 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-11T22:46:51.062 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-11T22:46:51.092 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-11T22:46:51.162 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:46:51.162 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-11T22:46:51.189 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-11T22:46:51.256 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-11T22:46:51.256 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-11T22:46:51.256 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/stdin-killer 2024-03-11T22:46:51.285 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-11T22:46:51.354 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:46:51.354 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/stdin-killer 2024-03-11T22:46:51.384 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-11T22:46:51.452 INFO:teuthology.run_tasks:Running task ceph... 2024-03-11T22:46:51.710 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-11T22:46:51.710 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 777 /var/log/ceph 2024-03-11T22:46:51.713 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 777 /var/log/ceph 2024-03-11T22:46:51.741 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-11T22:46:51.741 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-11T22:46:51.784 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-11T22:46:51.812 INFO:tasks.ceph:Creating extra log directories... 2024-03-11T22:46:51.812 DEBUG:teuthology.orchestra.run.smithi078:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-11T22:46:51.855 DEBUG:teuthology.orchestra.run.smithi132:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-11T22:46:51.896 INFO:tasks.ceph:Creating ceph cluster cluster1... 2024-03-11T22:46:51.897 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-11T22:46:51.897 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dis-2024-03-11_22:26:48-rbd-wip-dis-testing-distro-default-smithi/7592296', 'branch': 'wip-dis-testing', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum}', 'email': 'idryomov@gmail.com', 'first_in_suite': False, 'job_id': '7592296', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'dis-2024-03-11_22:26:48-rbd-wip-dis-testing-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-dis-testing'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'b117a47aa0ad81d207fdd9ba852068f2f262fea8'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'b117a47aa0ad81d207fdd9ba852068f2f262fea8'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-dis-testing-qa', 'sha1': 'a4679433626e24f222f64f757c6ab31760167603'}}, 'owner': 'scheduled_dis@teuthology', 'priority': 70, 'repo': 'https://github.com/ceph/ceph-ci', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 5633, 'sha1': 'b117a47aa0ad81d207fdd9ba852068f2f262fea8', 'sleep_before_teardown': 0, 'subset': '81/99', 'suite': 'rbd', 'suite_branch': 'wip-dis-testing-qa', 'suite_path': '/home/teuthworker/src/github.com_idryomov_ceph_a4679433626e24f222f64f757c6ab31760167603/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/idryomov/ceph', 'suite_sha1': 'a4679433626e24f222f64f757c6ab31760167603', 'targets': {'smithi078.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBALYIufoA3dz1qKE8kADQ7x1+cDKs+cZ3+eoCp4Ma9/PkJRs0d6nG11XqINqKSmIeLmh1eTH34m/OIJCkaVKVzE=', 'smithi132.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPNf1BzILs3QC7X/vb7hhud5sQ79Ov/m2HPKnz8nSwAUei2BPhVJ2BlnaQ0g4BlXne1kwcu0uHG9hKemdzGeUQw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'b117a47aa0ad81d207fdd9ba852068f2f262fea8'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'b117a47aa0ad81d207fdd9ba852068f2f262fea8'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-11_22:26:48', 'tube': 'smithi', 'user': 'dis', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-11T22:46:51.897 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-03-11T22:46:51.926 DEBUG:teuthology.orchestra.run.smithi132:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-03-11T22:46:51.940 DEBUG:teuthology.orchestra.run.smithi078:> sudo install -d -m0777 -- /var/run/ceph 2024-03-11T22:46:51.982 DEBUG:teuthology.orchestra.run.smithi132:> sudo install -d -m0777 -- /var/run/ceph 2024-03-11T22:46:52.009 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-11T22:46:52.009 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T22:46:52.066 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T22:46:52.066 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_1 2024-03-11T22:46:52.122 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T22:46:52.122 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:46:52.122 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-11T22:46:52.122 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:46:52.122 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:46:52.122 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-03-11 22:46:32.543856198 +0000 2024-03-11T22:46:52.122 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-03-11 22:43:56.651379296 +0000 2024-03-11T22:46:52.122 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-03-11 22:43:56.651379296 +0000 2024-03-11T22:46:52.123 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-03-11 22:43:56.651379296 +0000 2024-03-11T22:46:52.123 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T22:46:52.189 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-03-11T22:46:52.189 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-03-11T22:46:52.189 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000191736 s, 2.7 MB/s 2024-03-11T22:46:52.190 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T22:46:52.246 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_2 2024-03-11T22:46:52.301 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T22:46:52.301 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:46:52.301 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-11T22:46:52.301 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:46:52.301 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:46:52.301 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-03-11 22:46:32.543856198 +0000 2024-03-11T22:46:52.301 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-03-11 22:43:57.005373212 +0000 2024-03-11T22:46:52.302 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-03-11 22:43:57.005373212 +0000 2024-03-11T22:46:52.302 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-03-11 22:43:57.005373212 +0000 2024-03-11T22:46:52.302 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T22:46:52.366 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-03-11T22:46:52.367 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-03-11T22:46:52.367 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000193559 s, 2.6 MB/s 2024-03-11T22:46:52.368 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T22:46:52.424 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_3 2024-03-11T22:46:52.479 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T22:46:52.479 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:46:52.479 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-11T22:46:52.479 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:46:52.479 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:46:52.479 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-03-11 22:46:32.544856181 +0000 2024-03-11T22:46:52.479 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-03-11 22:43:57.355367197 +0000 2024-03-11T22:46:52.479 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-03-11 22:43:57.355367197 +0000 2024-03-11T22:46:52.479 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-03-11 22:43:57.355367197 +0000 2024-03-11T22:46:52.480 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T22:46:52.544 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-03-11T22:46:52.544 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-03-11T22:46:52.544 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000210031 s, 2.4 MB/s 2024-03-11T22:46:52.545 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T22:46:52.601 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_4 2024-03-11T22:46:52.660 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T22:46:52.660 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:46:52.660 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-11T22:46:52.660 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:46:52.661 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:46:52.661 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-03-11 22:46:32.544856181 +0000 2024-03-11T22:46:52.661 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-03-11 22:43:57.706361165 +0000 2024-03-11T22:46:52.661 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-03-11 22:43:57.706361165 +0000 2024-03-11T22:46:52.661 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-03-11 22:43:57.706361165 +0000 2024-03-11T22:46:52.661 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T22:46:52.725 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-03-11T22:46:52.725 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-03-11T22:46:52.726 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.00021019 s, 2.4 MB/s 2024-03-11T22:46:52.727 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T22:46:52.782 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-03-11T22:46:52.782 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi078.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-03-11T22:46:52.783 INFO:tasks.ceph:Generating config... 2024-03-11T22:46:52.791 INFO:tasks.ceph:[client] debug rbd = 20 2024-03-11T22:46:52.791 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-03-11T22:46:52.791 INFO:tasks.ceph:[client] log to stderr = False 2024-03-11T22:46:52.791 INFO:tasks.ceph:[client] rbd default features = 125 2024-03-11T22:46:52.791 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2024-03-11T22:46:52.792 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-11T22:46:52.792 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-11T22:46:52.792 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-11T22:46:52.792 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-11T22:46:52.792 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-11T22:46:52.792 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-11T22:46:52.792 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-11T22:46:52.792 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-11T22:46:52.792 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-11T22:46:52.792 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-03-11T22:46:52.792 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-11T22:46:52.792 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-11T22:46:52.792 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-11T22:46:52.792 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-11T22:46:52.793 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-11T22:46:52.793 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-11T22:46:52.793 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-11T22:46:52.793 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-11T22:46:52.793 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-11T22:46:52.793 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-11T22:46:52.793 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-11T22:46:52.793 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-03-11T22:46:52.793 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-03-11T22:46:52.793 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-11T22:46:52.793 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-11T22:46:52.793 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-03-11T22:46:52.793 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-11T22:46:52.793 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-11T22:46:52.794 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-11T22:46:52.794 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-11T22:46:52.794 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-11T22:46:52.794 INFO:tasks.ceph:Setting up cluster1.mon.a... 2024-03-11T22:46:52.794 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2024-03-11T22:46:52.870 INFO:teuthology.orchestra.run.smithi078.stdout:creating /etc/ceph/cluster1.keyring 2024-03-11T22:46:52.873 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2024-03-11T22:46:52.917 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-03-11T22:46:52.984 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.78')] 2024-03-11T22:46:52.984 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.78', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-03-11T22:46:52.985 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-11T22:46:52.985 DEBUG:teuthology.orchestra.run.smithi078:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-11T22:46:53.039 DEBUG:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.78 --print /home/ubuntu/cephtest/cluster1.monmap 2024-03-11T22:46:53.117 INFO:teuthology.orchestra.run.smithi078.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2024-03-11T22:46:53.117 INFO:teuthology.orchestra.run.smithi078.stdout:monmaptool: generated fsid b6063557-edc9-4ae4-8432-61b8810d0c16 2024-03-11T22:46:53.117 INFO:teuthology.orchestra.run.smithi078.stdout:setting min_mon_release = quincy 2024-03-11T22:46:53.117 INFO:teuthology.orchestra.run.smithi078.stdout:epoch 0 2024-03-11T22:46:53.117 INFO:teuthology.orchestra.run.smithi078.stdout:fsid b6063557-edc9-4ae4-8432-61b8810d0c16 2024-03-11T22:46:53.117 INFO:teuthology.orchestra.run.smithi078.stdout:last_changed 2024-03-11T22:46:53.110782+0000 2024-03-11T22:46:53.117 INFO:teuthology.orchestra.run.smithi078.stdout:created 2024-03-11T22:46:53.110782+0000 2024-03-11T22:46:53.117 INFO:teuthology.orchestra.run.smithi078.stdout:min_mon_release 17 (quincy) 2024-03-11T22:46:53.118 INFO:teuthology.orchestra.run.smithi078.stdout:election_strategy: 1 2024-03-11T22:46:53.118 INFO:teuthology.orchestra.run.smithi078.stdout:0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-03-11T22:46:53.118 INFO:teuthology.orchestra.run.smithi078.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2024-03-11T22:46:53.119 DEBUG:teuthology.orchestra.run.smithi078:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-11T22:46:53.174 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID b6063557-edc9-4ae4-8432-61b8810d0c16... 2024-03-11T22:46:53.175 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-03-11T22:46:53.217 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-03-11T22:46:53.264 INFO:teuthology.orchestra.run.smithi078.stdout:[global] 2024-03-11T22:46:53.264 INFO:teuthology.orchestra.run.smithi078.stdout: chdir = "" 2024-03-11T22:46:53.264 INFO:teuthology.orchestra.run.smithi078.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-11T22:46:53.264 INFO:teuthology.orchestra.run.smithi078.stdout: auth supported = cephx 2024-03-11T22:46:53.264 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:53.265 INFO:teuthology.orchestra.run.smithi078.stdout: filestore xattr use omap = true 2024-03-11T22:46:53.265 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:53.265 INFO:teuthology.orchestra.run.smithi078.stdout: mon clock drift allowed = 1.000 2024-03-11T22:46:53.265 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:53.265 INFO:teuthology.orchestra.run.smithi078.stdout: osd crush chooseleaf type = 0 2024-03-11T22:46:53.265 INFO:teuthology.orchestra.run.smithi078.stdout: auth debug = true 2024-03-11T22:46:53.265 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:53.265 INFO:teuthology.orchestra.run.smithi078.stdout: ms die on old message = true 2024-03-11T22:46:53.265 INFO:teuthology.orchestra.run.smithi078.stdout: ms die on bug = true 2024-03-11T22:46:53.265 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:53.265 INFO:teuthology.orchestra.run.smithi078.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-11T22:46:53.265 INFO:teuthology.orchestra.run.smithi078.stdout: mon pg warn max object skew = 0 2024-03-11T22:46:53.265 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:53.265 INFO:teuthology.orchestra.run.smithi078.stdout: # disable pg_autoscaler by default for new pools 2024-03-11T22:46:53.266 INFO:teuthology.orchestra.run.smithi078.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-11T22:46:53.266 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:53.266 INFO:teuthology.orchestra.run.smithi078.stdout: osd pool default size = 2 2024-03-11T22:46:53.266 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:53.266 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd allow primary affinity = true 2024-03-11T22:46:53.266 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd allow pg remap = true 2024-03-11T22:46:53.266 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on legacy crush tunables = false 2024-03-11T22:46:53.266 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on crush straw calc version zero = false 2024-03-11T22:46:53.266 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on no sortbitwise = false 2024-03-11T22:46:53.266 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on osd down out interval zero = false 2024-03-11T22:46:53.266 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on too few osds = false 2024-03-11T22:46:53.266 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-11T22:46:53.266 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-11T22:46:53.266 INFO:teuthology.orchestra.run.smithi078.stdout: mon_allow_pool_size_one = true 2024-03-11T22:46:53.266 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:53.267 INFO:teuthology.orchestra.run.smithi078.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-11T22:46:53.267 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:53.267 INFO:teuthology.orchestra.run.smithi078.stdout: osd default data pool replay window = 5 2024-03-11T22:46:53.267 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:53.267 INFO:teuthology.orchestra.run.smithi078.stdout: mon allow pool delete = true 2024-03-11T22:46:53.267 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:53.267 INFO:teuthology.orchestra.run.smithi078.stdout: mon cluster log file level = debug 2024-03-11T22:46:53.267 INFO:teuthology.orchestra.run.smithi078.stdout: debug asserts on shutdown = true 2024-03-11T22:46:53.267 INFO:teuthology.orchestra.run.smithi078.stdout: mon health detail to clog = false 2024-03-11T22:46:53.267 INFO:teuthology.orchestra.run.smithi078.stdout: mon host = 172.21.15.78 2024-03-11T22:46:53.267 INFO:teuthology.orchestra.run.smithi078.stdout: mon client directed command retry = 5 2024-03-11T22:46:53.267 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on pool no app = False 2024-03-11T22:46:53.267 INFO:teuthology.orchestra.run.smithi078.stdout: ms inject socket failures = 5000 2024-03-11T22:46:53.267 INFO:teuthology.orchestra.run.smithi078.stdout: fsid = b6063557-edc9-4ae4-8432-61b8810d0c16 2024-03-11T22:46:53.267 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:53.268 INFO:teuthology.orchestra.run.smithi078.stdout:[osd] 2024-03-11T22:46:53.268 INFO:teuthology.orchestra.run.smithi078.stdout: osd journal size = 100 2024-03-11T22:46:53.268 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:53.268 INFO:teuthology.orchestra.run.smithi078.stdout: osd scrub load threshold = 5.0 2024-03-11T22:46:53.268 INFO:teuthology.orchestra.run.smithi078.stdout: osd scrub max interval = 600 2024-03-11T22:46:53.268 INFO:teuthology.orchestra.run.smithi078.stdout: osd mclock profile = high_recovery_ops 2024-03-11T22:46:53.269 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:53.269 INFO:teuthology.orchestra.run.smithi078.stdout: osd recover clone overlap = true 2024-03-11T22:46:53.269 INFO:teuthology.orchestra.run.smithi078.stdout: osd recovery max chunk = 1048576 2024-03-11T22:46:53.269 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:53.269 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug shutdown = true 2024-03-11T22:46:53.269 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug op order = true 2024-03-11T22:46:53.269 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug verify stray on activate = true 2024-03-11T22:46:53.269 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug trim objects = true 2024-03-11T22:46:53.269 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:53.269 INFO:teuthology.orchestra.run.smithi078.stdout: osd open classes on start = true 2024-03-11T22:46:53.269 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug pg log writeout = true 2024-03-11T22:46:53.269 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:53.269 INFO:teuthology.orchestra.run.smithi078.stdout: osd deep scrub update digest min age = 30 2024-03-11T22:46:53.270 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:53.270 INFO:teuthology.orchestra.run.smithi078.stdout: osd map max advance = 10 2024-03-11T22:46:53.270 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:53.270 INFO:teuthology.orchestra.run.smithi078.stdout: journal zero on create = true 2024-03-11T22:46:53.270 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:53.270 INFO:teuthology.orchestra.run.smithi078.stdout: filestore ondisk finisher threads = 3 2024-03-11T22:46:53.270 INFO:teuthology.orchestra.run.smithi078.stdout: filestore apply finisher threads = 3 2024-03-11T22:46:53.270 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:53.270 INFO:teuthology.orchestra.run.smithi078.stdout: bdev debug aio = true 2024-03-11T22:46:53.270 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug misdirected ops = true 2024-03-11T22:46:53.270 INFO:teuthology.orchestra.run.smithi078.stdout: bdev async discard = True 2024-03-11T22:46:53.270 INFO:teuthology.orchestra.run.smithi078.stdout: bdev enable discard = True 2024-03-11T22:46:53.270 INFO:teuthology.orchestra.run.smithi078.stdout: bluestore allocator = stupid 2024-03-11T22:46:53.270 INFO:teuthology.orchestra.run.smithi078.stdout: bluestore block size = 96636764160 2024-03-11T22:46:53.270 INFO:teuthology.orchestra.run.smithi078.stdout: bluestore fsck on mount = True 2024-03-11T22:46:53.271 INFO:teuthology.orchestra.run.smithi078.stdout: debug bluefs = 1/20 2024-03-11T22:46:53.271 INFO:teuthology.orchestra.run.smithi078.stdout: debug bluestore = 1/20 2024-03-11T22:46:53.271 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2024-03-11T22:46:53.271 INFO:teuthology.orchestra.run.smithi078.stdout: debug osd = 20 2024-03-11T22:46:53.271 INFO:teuthology.orchestra.run.smithi078.stdout: debug rocksdb = 4/10 2024-03-11T22:46:53.271 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd backfillfull_ratio = 0.85 2024-03-11T22:46:53.271 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd full ratio = 0.9 2024-03-11T22:46:53.271 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd nearfull ratio = 0.8 2024-03-11T22:46:53.271 INFO:teuthology.orchestra.run.smithi078.stdout: osd failsafe full ratio = 0.95 2024-03-11T22:46:53.271 INFO:teuthology.orchestra.run.smithi078.stdout: osd objectstore = bluestore 2024-03-11T22:46:53.271 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:53.271 INFO:teuthology.orchestra.run.smithi078.stdout:[mgr] 2024-03-11T22:46:53.271 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2024-03-11T22:46:53.271 INFO:teuthology.orchestra.run.smithi078.stdout: debug mgr = 20 2024-03-11T22:46:53.271 INFO:teuthology.orchestra.run.smithi078.stdout: debug mon = 20 2024-03-11T22:46:53.272 INFO:teuthology.orchestra.run.smithi078.stdout: debug auth = 20 2024-03-11T22:46:53.272 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min pgs per osd = 4 2024-03-11T22:46:53.272 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min bytes per osd = 10 2024-03-11T22:46:53.272 INFO:teuthology.orchestra.run.smithi078.stdout: mgr/telemetry/nag = false 2024-03-11T22:46:53.272 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:53.272 INFO:teuthology.orchestra.run.smithi078.stdout:[mon] 2024-03-11T22:46:53.272 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2024-03-11T22:46:53.272 INFO:teuthology.orchestra.run.smithi078.stdout: debug mon = 20 2024-03-11T22:46:53.272 INFO:teuthology.orchestra.run.smithi078.stdout: debug paxos = 20 2024-03-11T22:46:53.272 INFO:teuthology.orchestra.run.smithi078.stdout: debug auth = 20 2024-03-11T22:46:53.272 INFO:teuthology.orchestra.run.smithi078.stdout: mon data avail warn = 5 2024-03-11T22:46:53.273 INFO:teuthology.orchestra.run.smithi078.stdout: mon mgr mkfs grace = 240 2024-03-11T22:46:53.273 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min pgs per osd = 4 2024-03-11T22:46:53.273 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd reporter subtree level = osd 2024-03-11T22:46:53.273 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd prime pg temp = true 2024-03-11T22:46:53.273 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min bytes per osd = 10 2024-03-11T22:46:53.273 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:53.273 INFO:teuthology.orchestra.run.smithi078.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-11T22:46:53.273 INFO:teuthology.orchestra.run.smithi078.stdout: auth mon ticket ttl = 660 # 11m 2024-03-11T22:46:53.273 INFO:teuthology.orchestra.run.smithi078.stdout: auth service ticket ttl = 240 # 4m 2024-03-11T22:46:53.274 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:53.274 INFO:teuthology.orchestra.run.smithi078.stdout: # don't complain about insecure global_id in the test suite 2024-03-11T22:46:53.274 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-11T22:46:53.274 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-11T22:46:53.274 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:53.274 INFO:teuthology.orchestra.run.smithi078.stdout: # 1m isn't quite enough 2024-03-11T22:46:53.274 INFO:teuthology.orchestra.run.smithi078.stdout: mon_down_mkfs_grace = 2m 2024-03-11T22:46:53.274 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:53.274 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_filestore_osds = false 2024-03-11T22:46:53.274 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:46:53.275 INFO:teuthology.orchestra.run.smithi078.stdout:[client] 2024-03-11T22:46:53.275 INFO:teuthology.orchestra.run.smithi078.stdout: rgw cache enabled = true 2024-03-11T22:46:53.275 INFO:teuthology.orchestra.run.smithi078.stdout: rgw enable ops log = true 2024-03-11T22:46:53.275 INFO:teuthology.orchestra.run.smithi078.stdout: rgw enable usage log = true 2024-03-11T22:46:53.275 INFO:teuthology.orchestra.run.smithi078.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-11T22:46:53.275 INFO:teuthology.orchestra.run.smithi078.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-11T22:46:53.275 INFO:teuthology.orchestra.run.smithi078.stdout: debug rbd = 20 2024-03-11T22:46:53.275 INFO:teuthology.orchestra.run.smithi078.stdout: debug rbd_mirror = 15 2024-03-11T22:46:53.275 INFO:teuthology.orchestra.run.smithi078.stdout: log to stderr = False 2024-03-11T22:46:53.275 INFO:teuthology.orchestra.run.smithi078.stdout: rbd default features = 125 2024-03-11T22:46:53.275 INFO:teuthology.orchestra.run.smithi078.stdout: rbd mirror image policy type = simple 2024-03-11T22:46:53.275 INFO:teuthology.orchestra.run.smithi078.stdout:[mon.a] 2024-03-11T22:46:53.275 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror.0] 2024-03-11T22:46:53.276 INFO:teuthology.orchestra.run.smithi078.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-11T22:46:53.276 INFO:teuthology.orchestra.run.smithi078.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-11T22:46:53.276 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror.1] 2024-03-11T22:46:53.276 INFO:teuthology.orchestra.run.smithi078.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-11T22:46:53.276 INFO:teuthology.orchestra.run.smithi078.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-11T22:46:53.276 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror.2] 2024-03-11T22:46:53.276 INFO:teuthology.orchestra.run.smithi078.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-11T22:46:53.276 INFO:teuthology.orchestra.run.smithi078.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-11T22:46:53.276 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror.3] 2024-03-11T22:46:53.276 INFO:teuthology.orchestra.run.smithi078.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-11T22:46:53.276 INFO:teuthology.orchestra.run.smithi078.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-11T22:46:53.277 INFO:teuthology.orchestra.run.smithi132.stdout:[global] 2024-03-11T22:46:53.277 INFO:teuthology.orchestra.run.smithi132.stdout: chdir = "" 2024-03-11T22:46:53.277 INFO:teuthology.orchestra.run.smithi132.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-11T22:46:53.277 INFO:teuthology.orchestra.run.smithi132.stdout: auth supported = cephx 2024-03-11T22:46:53.278 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:53.278 INFO:teuthology.orchestra.run.smithi132.stdout: filestore xattr use omap = true 2024-03-11T22:46:53.278 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:53.278 INFO:teuthology.orchestra.run.smithi132.stdout: mon clock drift allowed = 1.000 2024-03-11T22:46:53.278 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:53.278 INFO:teuthology.orchestra.run.smithi132.stdout: osd crush chooseleaf type = 0 2024-03-11T22:46:53.278 INFO:teuthology.orchestra.run.smithi132.stdout: auth debug = true 2024-03-11T22:46:53.278 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:53.278 INFO:teuthology.orchestra.run.smithi132.stdout: ms die on old message = true 2024-03-11T22:46:53.278 INFO:teuthology.orchestra.run.smithi132.stdout: ms die on bug = true 2024-03-11T22:46:53.278 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:53.278 INFO:teuthology.orchestra.run.smithi132.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-11T22:46:53.279 INFO:teuthology.orchestra.run.smithi132.stdout: mon pg warn max object skew = 0 2024-03-11T22:46:53.279 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:53.280 INFO:teuthology.orchestra.run.smithi132.stdout: # disable pg_autoscaler by default for new pools 2024-03-11T22:46:53.280 INFO:teuthology.orchestra.run.smithi132.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-11T22:46:53.280 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:53.280 INFO:teuthology.orchestra.run.smithi132.stdout: osd pool default size = 2 2024-03-11T22:46:53.280 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:53.280 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd allow primary affinity = true 2024-03-11T22:46:53.280 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd allow pg remap = true 2024-03-11T22:46:53.280 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on legacy crush tunables = false 2024-03-11T22:46:53.280 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on crush straw calc version zero = false 2024-03-11T22:46:53.280 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on no sortbitwise = false 2024-03-11T22:46:53.280 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on osd down out interval zero = false 2024-03-11T22:46:53.280 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on too few osds = false 2024-03-11T22:46:53.281 INFO:teuthology.orchestra.run.smithi132.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-11T22:46:53.281 INFO:teuthology.orchestra.run.smithi132.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-11T22:46:53.281 INFO:teuthology.orchestra.run.smithi132.stdout: mon_allow_pool_size_one = true 2024-03-11T22:46:53.281 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:53.281 INFO:teuthology.orchestra.run.smithi132.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-11T22:46:53.281 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:53.281 INFO:teuthology.orchestra.run.smithi132.stdout: osd default data pool replay window = 5 2024-03-11T22:46:53.281 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:53.281 INFO:teuthology.orchestra.run.smithi132.stdout: mon allow pool delete = true 2024-03-11T22:46:53.281 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:53.281 INFO:teuthology.orchestra.run.smithi132.stdout: mon cluster log file level = debug 2024-03-11T22:46:53.281 INFO:teuthology.orchestra.run.smithi132.stdout: debug asserts on shutdown = true 2024-03-11T22:46:53.281 INFO:teuthology.orchestra.run.smithi132.stdout: mon health detail to clog = false 2024-03-11T22:46:53.281 INFO:teuthology.orchestra.run.smithi132.stdout: mon host = 172.21.15.78 2024-03-11T22:46:53.281 INFO:teuthology.orchestra.run.smithi132.stdout: mon client directed command retry = 5 2024-03-11T22:46:53.282 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on pool no app = False 2024-03-11T22:46:53.282 INFO:teuthology.orchestra.run.smithi132.stdout: ms inject socket failures = 5000 2024-03-11T22:46:53.282 INFO:teuthology.orchestra.run.smithi132.stdout: fsid = b6063557-edc9-4ae4-8432-61b8810d0c16 2024-03-11T22:46:53.282 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:53.282 INFO:teuthology.orchestra.run.smithi132.stdout:[osd] 2024-03-11T22:46:53.282 INFO:teuthology.orchestra.run.smithi132.stdout: osd journal size = 100 2024-03-11T22:46:53.282 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:53.282 INFO:teuthology.orchestra.run.smithi132.stdout: osd scrub load threshold = 5.0 2024-03-11T22:46:53.282 INFO:teuthology.orchestra.run.smithi132.stdout: osd scrub max interval = 600 2024-03-11T22:46:53.282 INFO:teuthology.orchestra.run.smithi132.stdout: osd mclock profile = high_recovery_ops 2024-03-11T22:46:53.282 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:53.282 INFO:teuthology.orchestra.run.smithi132.stdout: osd recover clone overlap = true 2024-03-11T22:46:53.283 INFO:teuthology.orchestra.run.smithi132.stdout: osd recovery max chunk = 1048576 2024-03-11T22:46:53.283 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:53.283 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug shutdown = true 2024-03-11T22:46:53.283 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug op order = true 2024-03-11T22:46:53.283 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug verify stray on activate = true 2024-03-11T22:46:53.283 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug trim objects = true 2024-03-11T22:46:53.283 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:53.283 INFO:teuthology.orchestra.run.smithi132.stdout: osd open classes on start = true 2024-03-11T22:46:53.283 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug pg log writeout = true 2024-03-11T22:46:53.283 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:53.283 INFO:teuthology.orchestra.run.smithi132.stdout: osd deep scrub update digest min age = 30 2024-03-11T22:46:53.283 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:53.283 INFO:teuthology.orchestra.run.smithi132.stdout: osd map max advance = 10 2024-03-11T22:46:53.283 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:53.283 INFO:teuthology.orchestra.run.smithi132.stdout: journal zero on create = true 2024-03-11T22:46:53.284 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:53.284 INFO:teuthology.orchestra.run.smithi132.stdout: filestore ondisk finisher threads = 3 2024-03-11T22:46:53.284 INFO:teuthology.orchestra.run.smithi132.stdout: filestore apply finisher threads = 3 2024-03-11T22:46:53.284 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:53.284 INFO:teuthology.orchestra.run.smithi132.stdout: bdev debug aio = true 2024-03-11T22:46:53.284 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug misdirected ops = true 2024-03-11T22:46:53.284 INFO:teuthology.orchestra.run.smithi132.stdout: bdev async discard = True 2024-03-11T22:46:53.284 INFO:teuthology.orchestra.run.smithi132.stdout: bdev enable discard = True 2024-03-11T22:46:53.284 INFO:teuthology.orchestra.run.smithi132.stdout: bluestore allocator = stupid 2024-03-11T22:46:53.284 INFO:teuthology.orchestra.run.smithi132.stdout: bluestore block size = 96636764160 2024-03-11T22:46:53.284 INFO:teuthology.orchestra.run.smithi132.stdout: bluestore fsck on mount = True 2024-03-11T22:46:53.284 INFO:teuthology.orchestra.run.smithi132.stdout: debug bluefs = 1/20 2024-03-11T22:46:53.284 INFO:teuthology.orchestra.run.smithi132.stdout: debug bluestore = 1/20 2024-03-11T22:46:53.284 INFO:teuthology.orchestra.run.smithi132.stdout: debug ms = 1 2024-03-11T22:46:53.285 INFO:teuthology.orchestra.run.smithi132.stdout: debug osd = 20 2024-03-11T22:46:53.285 INFO:teuthology.orchestra.run.smithi132.stdout: debug rocksdb = 4/10 2024-03-11T22:46:53.285 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd backfillfull_ratio = 0.85 2024-03-11T22:46:53.285 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd full ratio = 0.9 2024-03-11T22:46:53.285 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd nearfull ratio = 0.8 2024-03-11T22:46:53.285 INFO:teuthology.orchestra.run.smithi132.stdout: osd failsafe full ratio = 0.95 2024-03-11T22:46:53.285 INFO:teuthology.orchestra.run.smithi132.stdout: osd objectstore = bluestore 2024-03-11T22:46:53.285 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:53.285 INFO:teuthology.orchestra.run.smithi132.stdout:[mgr] 2024-03-11T22:46:53.285 INFO:teuthology.orchestra.run.smithi132.stdout: debug ms = 1 2024-03-11T22:46:53.285 INFO:teuthology.orchestra.run.smithi132.stdout: debug mgr = 20 2024-03-11T22:46:53.285 INFO:teuthology.orchestra.run.smithi132.stdout: debug mon = 20 2024-03-11T22:46:53.285 INFO:teuthology.orchestra.run.smithi132.stdout: debug auth = 20 2024-03-11T22:46:53.285 INFO:teuthology.orchestra.run.smithi132.stdout: mon reweight min pgs per osd = 4 2024-03-11T22:46:53.285 INFO:teuthology.orchestra.run.smithi132.stdout: mon reweight min bytes per osd = 10 2024-03-11T22:46:53.286 INFO:teuthology.orchestra.run.smithi132.stdout: mgr/telemetry/nag = false 2024-03-11T22:46:53.286 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:53.286 INFO:teuthology.orchestra.run.smithi132.stdout:[mon] 2024-03-11T22:46:53.286 INFO:teuthology.orchestra.run.smithi132.stdout: debug ms = 1 2024-03-11T22:46:53.286 INFO:teuthology.orchestra.run.smithi132.stdout: debug mon = 20 2024-03-11T22:46:53.286 INFO:teuthology.orchestra.run.smithi132.stdout: debug paxos = 20 2024-03-11T22:46:53.286 INFO:teuthology.orchestra.run.smithi132.stdout: debug auth = 20 2024-03-11T22:46:53.286 INFO:teuthology.orchestra.run.smithi132.stdout: mon data avail warn = 5 2024-03-11T22:46:53.286 INFO:teuthology.orchestra.run.smithi132.stdout: mon mgr mkfs grace = 240 2024-03-11T22:46:53.286 INFO:teuthology.orchestra.run.smithi132.stdout: mon reweight min pgs per osd = 4 2024-03-11T22:46:53.286 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd reporter subtree level = osd 2024-03-11T22:46:53.286 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd prime pg temp = true 2024-03-11T22:46:53.286 INFO:teuthology.orchestra.run.smithi132.stdout: mon reweight min bytes per osd = 10 2024-03-11T22:46:53.286 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:53.286 INFO:teuthology.orchestra.run.smithi132.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-11T22:46:53.287 INFO:teuthology.orchestra.run.smithi132.stdout: auth mon ticket ttl = 660 # 11m 2024-03-11T22:46:53.287 INFO:teuthology.orchestra.run.smithi132.stdout: auth service ticket ttl = 240 # 4m 2024-03-11T22:46:53.287 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:53.287 INFO:teuthology.orchestra.run.smithi132.stdout: # don't complain about insecure global_id in the test suite 2024-03-11T22:46:53.287 INFO:teuthology.orchestra.run.smithi132.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-11T22:46:53.287 INFO:teuthology.orchestra.run.smithi132.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-11T22:46:53.287 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:53.287 INFO:teuthology.orchestra.run.smithi132.stdout: # 1m isn't quite enough 2024-03-11T22:46:53.287 INFO:teuthology.orchestra.run.smithi132.stdout: mon_down_mkfs_grace = 2m 2024-03-11T22:46:53.287 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:53.287 INFO:teuthology.orchestra.run.smithi132.stdout: mon_warn_on_filestore_osds = false 2024-03-11T22:46:53.287 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:46:53.287 INFO:teuthology.orchestra.run.smithi132.stdout:[client] 2024-03-11T22:46:53.287 INFO:teuthology.orchestra.run.smithi132.stdout: rgw cache enabled = true 2024-03-11T22:46:53.287 INFO:teuthology.orchestra.run.smithi132.stdout: rgw enable ops log = true 2024-03-11T22:46:53.288 INFO:teuthology.orchestra.run.smithi132.stdout: rgw enable usage log = true 2024-03-11T22:46:53.288 INFO:teuthology.orchestra.run.smithi132.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-11T22:46:53.288 INFO:teuthology.orchestra.run.smithi132.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-11T22:46:53.288 INFO:teuthology.orchestra.run.smithi132.stdout: debug rbd = 20 2024-03-11T22:46:53.288 INFO:teuthology.orchestra.run.smithi132.stdout: debug rbd_mirror = 15 2024-03-11T22:46:53.288 INFO:teuthology.orchestra.run.smithi132.stdout: log to stderr = False 2024-03-11T22:46:53.288 INFO:teuthology.orchestra.run.smithi132.stdout: rbd default features = 125 2024-03-11T22:46:53.288 INFO:teuthology.orchestra.run.smithi132.stdout: rbd mirror image policy type = simple 2024-03-11T22:46:53.288 INFO:teuthology.orchestra.run.smithi132.stdout:[mon.a] 2024-03-11T22:46:53.288 INFO:teuthology.orchestra.run.smithi132.stdout:[client.mirror.0] 2024-03-11T22:46:53.288 INFO:teuthology.orchestra.run.smithi132.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-11T22:46:53.288 INFO:teuthology.orchestra.run.smithi132.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-11T22:46:53.288 INFO:teuthology.orchestra.run.smithi132.stdout:[client.mirror.1] 2024-03-11T22:46:53.288 INFO:teuthology.orchestra.run.smithi132.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-11T22:46:53.288 INFO:teuthology.orchestra.run.smithi132.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-11T22:46:53.289 INFO:teuthology.orchestra.run.smithi132.stdout:[client.mirror.2] 2024-03-11T22:46:53.289 INFO:teuthology.orchestra.run.smithi132.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-11T22:46:53.289 INFO:teuthology.orchestra.run.smithi132.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-11T22:46:53.289 INFO:teuthology.orchestra.run.smithi132.stdout:[client.mirror.3] 2024-03-11T22:46:53.289 INFO:teuthology.orchestra.run.smithi132.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-11T22:46:53.289 INFO:teuthology.orchestra.run.smithi132.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-11T22:46:53.291 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2024-03-11T22:46:53.291 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2024-03-11T22:46:53.386 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-11T22:46:53.386 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-11T22:46:53.387 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2024-03-11T22:46:53.401 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-11T22:46:53.401 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2024-03-11T22:46:53.455 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi078.front.sepia.ceph.com 2024-03-11T22:46:53.456 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-11T22:46:53.456 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/cluster1.keyring 2024-03-11T22:46:53.456 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-03-11T22:46:53.536 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-11T22:46:53.536 DEBUG:teuthology.orchestra.run.smithi078:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-03-11T22:46:53.590 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi132.front.sepia.ceph.com 2024-03-11T22:46:53.590 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:46:53.590 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/ceph/cluster1.keyring 2024-03-11T22:46:53.590 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-03-11T22:46:53.632 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:46:53.632 DEBUG:teuthology.orchestra.run.smithi132:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-03-11T22:46:53.688 INFO:tasks.ceph:Setting up mon nodes... 2024-03-11T22:46:53.688 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-11T22:46:53.688 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2024-03-11T22:46:53.742 INFO:teuthology.orchestra.run.smithi078.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2024-03-11T22:46:53.745 INFO:tasks.ceph:Setting up mds nodes... 2024-03-11T22:46:53.745 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-11T22:46:53.745 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2024-03-11T22:46:53.787 INFO:teuthology.orchestra.run.smithi078.stdout:creating /etc/ceph/cluster1.client.0.keyring 2024-03-11T22:46:53.802 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2024-03-11T22:46:53.858 INFO:teuthology.orchestra.run.smithi132.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2024-03-11T22:46:53.873 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2024-03-11T22:46:53.916 INFO:teuthology.orchestra.run.smithi132.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2024-03-11T22:46:53.931 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2024-03-11T22:46:54.012 INFO:teuthology.orchestra.run.smithi132.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2024-03-11T22:46:54.028 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2024-03-11T22:46:54.110 INFO:teuthology.orchestra.run.smithi132.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2024-03-11T22:46:54.126 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2024-03-11T22:46:54.208 INFO:teuthology.orchestra.run.smithi132.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2024-03-11T22:46:54.224 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2024-03-11T22:46:54.307 INFO:teuthology.orchestra.run.smithi132.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2024-03-11T22:46:54.322 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2024-03-11T22:46:54.404 INFO:teuthology.orchestra.run.smithi132.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2024-03-11T22:46:54.420 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2024-03-11T22:46:54.503 INFO:teuthology.orchestra.run.smithi132.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2024-03-11T22:46:54.519 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-11T22:46:54.519 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi078.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-03-11T22:46:54.519 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2024-03-11T22:46:54.547 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-03-11T22:46:54.547 INFO:tasks.ceph:role: cluster1.osd.0 2024-03-11T22:46:54.547 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi078.front.sepia.ceph.com 2024-03-11T22:46:54.547 DEBUG:teuthology.orchestra.run.smithi078:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-11T22:46:54.622 INFO:teuthology.orchestra.run.smithi078.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-11T22:46:54.623 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-11T22:46:54.623 INFO:teuthology.orchestra.run.smithi078.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-11T22:46:54.623 INFO:teuthology.orchestra.run.smithi078.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-11T22:46:54.623 INFO:teuthology.orchestra.run.smithi078.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-11T22:46:54.623 INFO:teuthology.orchestra.run.smithi078.stdout: = sunit=0 swidth=0 blks 2024-03-11T22:46:54.623 INFO:teuthology.orchestra.run.smithi078.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-11T22:46:54.623 INFO:teuthology.orchestra.run.smithi078.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-11T22:46:54.623 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-11T22:46:54.623 INFO:teuthology.orchestra.run.smithi078.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-11T22:46:55.131 INFO:teuthology.orchestra.run.smithi078.stdout:Discarding blocks...Done. 2024-03-11T22:46:55.134 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi078.front.sepia.ceph.com -o noatime 2024-03-11T22:46:55.134 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2024-03-11T22:46:55.268 DEBUG:teuthology.orchestra.run.smithi078:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2024-03-11T22:46:55.308 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2024-03-11T22:46:55.377 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-03-11T22:46:55.377 INFO:tasks.ceph:role: cluster1.osd.1 2024-03-11T22:46:55.377 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi078.front.sepia.ceph.com 2024-03-11T22:46:55.378 DEBUG:teuthology.orchestra.run.smithi078:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-11T22:46:55.453 INFO:teuthology.orchestra.run.smithi078.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-11T22:46:55.453 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-11T22:46:55.453 INFO:teuthology.orchestra.run.smithi078.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-11T22:46:55.453 INFO:teuthology.orchestra.run.smithi078.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-11T22:46:55.453 INFO:teuthology.orchestra.run.smithi078.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-11T22:46:55.453 INFO:teuthology.orchestra.run.smithi078.stdout: = sunit=0 swidth=0 blks 2024-03-11T22:46:55.454 INFO:teuthology.orchestra.run.smithi078.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-11T22:46:55.454 INFO:teuthology.orchestra.run.smithi078.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-11T22:46:55.454 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-11T22:46:55.454 INFO:teuthology.orchestra.run.smithi078.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-11T22:46:55.919 INFO:teuthology.orchestra.run.smithi078.stdout:Discarding blocks...Done. 2024-03-11T22:46:55.919 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi078.front.sepia.ceph.com -o noatime 2024-03-11T22:46:55.920 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2024-03-11T22:46:56.013 DEBUG:teuthology.orchestra.run.smithi078:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2024-03-11T22:46:56.044 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2024-03-11T22:46:56.112 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-03-11T22:46:56.112 INFO:tasks.ceph:role: cluster1.osd.2 2024-03-11T22:46:56.112 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi078.front.sepia.ceph.com 2024-03-11T22:46:56.112 DEBUG:teuthology.orchestra.run.smithi078:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-11T22:46:56.186 INFO:teuthology.orchestra.run.smithi078.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-11T22:46:56.186 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-11T22:46:56.186 INFO:teuthology.orchestra.run.smithi078.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-11T22:46:56.186 INFO:teuthology.orchestra.run.smithi078.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-11T22:46:56.186 INFO:teuthology.orchestra.run.smithi078.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-11T22:46:56.186 INFO:teuthology.orchestra.run.smithi078.stdout: = sunit=0 swidth=0 blks 2024-03-11T22:46:56.187 INFO:teuthology.orchestra.run.smithi078.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-11T22:46:56.187 INFO:teuthology.orchestra.run.smithi078.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-11T22:46:56.187 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-11T22:46:56.187 INFO:teuthology.orchestra.run.smithi078.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-11T22:46:56.675 INFO:teuthology.orchestra.run.smithi078.stdout:Discarding blocks...Done. 2024-03-11T22:46:56.677 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi078.front.sepia.ceph.com -o noatime 2024-03-11T22:46:56.677 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2024-03-11T22:46:56.800 DEBUG:teuthology.orchestra.run.smithi078:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2024-03-11T22:46:56.832 DEBUG:teuthology.orchestra.run.smithi078:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-03-11T22:46:56.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-11T22:46:56.923+0000 7fb9a2019640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2024-03-11T22:46:56.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-11T22:46:56.923+0000 7fb9a2019640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2024-03-11T22:46:56.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-11T22:46:56.923+0000 7fb9a2019640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-03-11T22:46:56.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-11T22:46:56.923+0000 7fb9a2019640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-03-11T22:46:56.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-11T22:46:56.923+0000 7fb9a2019640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-03-11T22:46:56.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-11T22:46:56.931+0000 7fb9a2019640 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2024-03-11T22:46:59.790 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2024-03-11T22:46:59.823 DEBUG:teuthology.orchestra.run.smithi078:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-03-11T22:46:59.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-11T22:46:59.909+0000 7f70ef21e640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2024-03-11T22:46:59.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-11T22:46:59.909+0000 7f70ef21e640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2024-03-11T22:46:59.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-11T22:46:59.909+0000 7f70ef21e640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-03-11T22:46:59.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-11T22:46:59.909+0000 7f70ef21e640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-03-11T22:46:59.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-11T22:46:59.909+0000 7f70ef21e640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-03-11T22:46:59.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-11T22:46:59.910+0000 7f70ef21e640 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2024-03-11T22:47:02.766 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2024-03-11T22:47:02.798 DEBUG:teuthology.orchestra.run.smithi078:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-03-11T22:47:02.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-11T22:47:02.884+0000 7f0d3b80d640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2024-03-11T22:47:02.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-11T22:47:02.884+0000 7f0d3b80d640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2024-03-11T22:47:02.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-11T22:47:02.884+0000 7f0d3b80d640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-03-11T22:47:02.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-11T22:47:02.884+0000 7f0d3b80d640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-03-11T22:47:02.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-11T22:47:02.884+0000 7f0d3b80d640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-03-11T22:47:02.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-11T22:47:02.885+0000 7f0d3b80d640 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2024-03-11T22:47:05.741 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2024-03-11T22:47:05.771 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-11T22:47:05.771 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-11T22:47:05.771 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2024-03-11T22:47:05.841 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-11T22:47:05.841 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2024-03-11T22:47:05.910 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-11T22:47:05.910 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2024-03-11T22:47:05.978 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-11T22:47:05.978 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2024-03-11T22:47:06.045 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-11T22:47:06.045 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2024-03-11T22:47:06.102 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:47:06.102 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2024-03-11T22:47:06.121 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:47:06.121 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2024-03-11T22:47:06.179 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:47:06.179 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2024-03-11T22:47:06.237 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:47:06.237 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2024-03-11T22:47:06.295 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:47:06.295 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2024-03-11T22:47:06.352 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:47:06.352 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2024-03-11T22:47:06.409 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:47:06.409 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2024-03-11T22:47:06.466 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:47:06.466 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2024-03-11T22:47:06.522 INFO:tasks.ceph:Adding keys to all mons... 2024-03-11T22:47:06.523 DEBUG:teuthology.orchestra.run.smithi078:> sudo tee -a /etc/ceph/cluster1.keyring 2024-03-11T22:47:06.550 INFO:teuthology.orchestra.run.smithi078.stdout:[mgr.x] 2024-03-11T22:47:06.551 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDdie9lL5rmKxAA7QX+6uG+Nr8kpgODqN6CJw== 2024-03-11T22:47:06.551 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.0] 2024-03-11T22:47:06.551 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDgie9lmaoWNxAAwhQScK5acyWWFlFtqcMXSQ== 2024-03-11T22:47:06.551 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.1] 2024-03-11T22:47:06.551 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDjie9lqQRDNhAAFS5zqbLnZcTAX6HDELvPBg== 2024-03-11T22:47:06.551 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.2] 2024-03-11T22:47:06.551 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDmie9lwdXBNBAAA76d57YHN3qbYkibfRyfPQ== 2024-03-11T22:47:06.551 INFO:teuthology.orchestra.run.smithi078.stdout:[client.0] 2024-03-11T22:47:06.551 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDdie9lTzyPLhAAkIiBwNjx4PrtBQ+IlQD5MQ== 2024-03-11T22:47:06.552 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror] 2024-03-11T22:47:06.552 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDdie9lJ5nFMhAA+4V/28Td6xrE7+4tOe4JEw== 2024-03-11T22:47:06.552 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror.0] 2024-03-11T22:47:06.552 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDdie9l17QzNhAA1SWbNnege6pXuycsc0npdw== 2024-03-11T22:47:06.552 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror.1] 2024-03-11T22:47:06.552 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDeie9lNr9jABAAE1afSLUktMTAX0tVN3wH2w== 2024-03-11T22:47:06.552 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror.2] 2024-03-11T22:47:06.552 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDeie9liOw6BhAAKbMgwcmyT9Etrco1JFn0dQ== 2024-03-11T22:47:06.552 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror.3] 2024-03-11T22:47:06.552 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDeie9lA8oPDBAAdo3fTzAGL+ODxMwEqUBHEg== 2024-03-11T22:47:06.552 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror.4] 2024-03-11T22:47:06.552 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDeie9leSftERAADChtzaMYGrbJ/H6ba9vang== 2024-03-11T22:47:06.552 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror.5] 2024-03-11T22:47:06.552 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDeie9lZcC+FxAAN0GCUdbsX+n+eHGSEYrYxw== 2024-03-11T22:47:06.553 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror.6] 2024-03-11T22:47:06.553 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDeie9lveScHRAABcZwhSYaqIs1KfukIvc+wA== 2024-03-11T22:47:06.553 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-11T22:47:06.649 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-11T22:47:06.703 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-11T22:47:06.754 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-11T22:47:06.806 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-11T22:47:06.861 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-11T22:47:06.915 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-11T22:47:06.971 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-11T22:47:07.025 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-11T22:47:07.080 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-11T22:47:07.134 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-11T22:47:07.190 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-11T22:47:07.245 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-11T22:47:07.300 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-11T22:47:07.301 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2024-03-11T22:47:07.328 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2024-03-11T22:47:07.426 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2024-03-11T22:47:07.453 DEBUG:teuthology.orchestra.run.smithi078:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2024-03-11T22:47:07.508 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2024-03-11T22:47:07.508 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2024-03-11T22:47:07.508 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2024-03-11T22:47:07.550 INFO:tasks.ceph.cluster1.mon.a:Started 2024-03-11T22:47:07.550 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2024-03-11T22:47:07.551 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2024-03-11T22:47:07.551 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2024-03-11T22:47:07.553 INFO:tasks.ceph.cluster1.mgr.x:Started 2024-03-11T22:47:07.553 INFO:tasks.ceph:Setting crush tunables to default 2024-03-11T22:47:07.553 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --cluster cluster1 osd crush tunables default 2024-03-11T22:47:07.740 INFO:teuthology.orchestra.run.smithi078.stderr:adjusted tunables profile to default 2024-03-11T22:47:07.751 INFO:tasks.ceph:check_enable_crimson: False 2024-03-11T22:47:07.751 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2024-03-11T22:47:07.752 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-11T22:47:07.752 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2024-03-11T22:47:07.779 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-11T22:47:07.779 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2024-03-11T22:47:07.846 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-11T22:47:07.846 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2024-03-11T22:47:07.915 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --cluster cluster1 osd new 916b2b84-4c20-43d3-870e-890cfc5a42d6 0 2024-03-11T22:47:08.135 INFO:teuthology.orchestra.run.smithi078.stdout:0 2024-03-11T22:47:08.147 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --cluster cluster1 osd new 44230f7e-eaff-4b71-b1cc-867151169e60 1 2024-03-11T22:47:08.302 INFO:tasks.ceph.cluster1.mgr.x.smithi078.stderr:2024-03-11T22:47:08.295+0000 7f1798442040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T22:47:08.328 INFO:teuthology.orchestra.run.smithi078.stdout:1 2024-03-11T22:47:08.338 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --cluster cluster1 osd new 42db0bd8-b0b4-40e6-aa12-ecffa1c99359 2 2024-03-11T22:47:08.359 INFO:tasks.ceph.cluster1.mgr.x.smithi078.stderr:2024-03-11T22:47:08.352+0000 7f1798442040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T22:47:08.519 INFO:teuthology.orchestra.run.smithi078.stdout:2 2024-03-11T22:47:08.530 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2024-03-11T22:47:08.531 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2024-03-11T22:47:08.573 INFO:tasks.ceph.cluster1.osd.0:Started 2024-03-11T22:47:08.573 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2024-03-11T22:47:08.574 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2024-03-11T22:47:08.575 INFO:tasks.ceph.cluster1.osd.1:Started 2024-03-11T22:47:08.576 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2024-03-11T22:47:08.576 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2024-03-11T22:47:08.578 INFO:tasks.ceph.cluster1.osd.2:Started 2024-03-11T22:47:08.578 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-03-11T22:47:08.778 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:47:08.778 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":5,"fsid":"b6063557-edc9-4ae4-8432-61b8810d0c16","created":"2024-03-11T22:47:07.655058+0000","modified":"2024-03-11T22:47:08.512853+0000","last_up_change":"0.000000","last_in_change":"2024-03-11T22:47:08.512853+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"916b2b84-4c20-43d3-870e-890cfc5a42d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"44230f7e-eaff-4b71-b1cc-867151169e60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"42db0bd8-b0b4-40e6-aa12-ecffa1c99359","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T22:47:08.790 INFO:tasks.ceph.ceph_manager.cluster1:[] 2024-03-11T22:47:08.790 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-11T22:47:08.909 INFO:tasks.ceph.cluster1.mgr.x.smithi078.stderr:2024-03-11T22:47:08.901+0000 7f1798442040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T22:47:08.964 INFO:tasks.ceph.cluster1.mgr.x.smithi078.stderr:2024-03-11T22:47:08.956+0000 7f1798442040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T22:47:09.008 INFO:tasks.ceph.cluster1.mgr.x.smithi078.stderr:2024-03-11T22:47:09.001+0000 7f1798442040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T22:47:09.061 INFO:tasks.ceph.cluster1.mgr.x.smithi078.stderr:2024-03-11T22:47:09.054+0000 7f1798442040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T22:47:09.093 DEBUG:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-03-11T22:47:09.104 INFO:tasks.ceph.cluster1.mgr.x.smithi078.stderr:2024-03-11T22:47:09.097+0000 7f1798442040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T22:47:09.178 INFO:tasks.ceph.cluster1.osd.1.smithi078.stderr:2024-03-11T22:47:09.171+0000 7f0d9a41b640 -1 Falling back to public interface 2024-03-11T22:47:09.179 INFO:tasks.ceph.cluster1.osd.0.smithi078.stderr:2024-03-11T22:47:09.172+0000 7ff7bc010640 -1 Falling back to public interface 2024-03-11T22:47:09.198 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2024-03-11T22:47:09.191+0000 7ff0d216d640 -1 Falling back to public interface 2024-03-11T22:47:09.228 INFO:tasks.ceph.cluster1.mgr.x.smithi078.stderr:2024-03-11T22:47:09.221+0000 7f1798442040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T22:47:09.255 INFO:teuthology.misc.health.smithi078.stdout: 2024-03-11T22:47:09.255 INFO:teuthology.misc.health.smithi078.stdout:{"epoch":5,"fsid":"b6063557-edc9-4ae4-8432-61b8810d0c16","created":"2024-03-11T22:47:07.655058+0000","modified":"2024-03-11T22:47:08.512853+0000","last_up_change":"0.000000","last_in_change":"2024-03-11T22:47:08.512853+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"916b2b84-4c20-43d3-870e-890cfc5a42d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"44230f7e-eaff-4b71-b1cc-867151169e60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"42db0bd8-b0b4-40e6-aa12-ecffa1c99359","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T22:47:09.265 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-03-11T22:47:09.300 INFO:tasks.ceph.cluster1.mgr.x.smithi078.stderr:2024-03-11T22:47:09.293+0000 7f1798442040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T22:47:09.346 INFO:tasks.ceph.cluster1.mgr.x.smithi078.stderr:2024-03-11T22:47:09.338+0000 7f1798442040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T22:47:09.397 INFO:tasks.ceph.cluster1.mgr.x.smithi078.stderr:2024-03-11T22:47:09.390+0000 7f1798442040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T22:47:09.443 INFO:tasks.ceph.cluster1.mgr.x.smithi078.stderr:2024-03-11T22:47:09.436+0000 7f1798442040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T22:47:09.820 INFO:tasks.ceph.cluster1.mgr.x.smithi078.stderr:2024-03-11T22:47:09.813+0000 7f1798442040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T22:47:09.951 INFO:tasks.ceph.cluster1.mgr.x.smithi078.stderr:2024-03-11T22:47:09.944+0000 7f1798442040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T22:47:10.078 INFO:tasks.ceph.cluster1.mgr.x.smithi078.stderr:2024-03-11T22:47:10.071+0000 7f1798442040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T22:47:10.123 INFO:tasks.ceph.cluster1.mgr.x.smithi078.stderr:2024-03-11T22:47:10.116+0000 7f1798442040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T22:47:10.233 INFO:tasks.ceph.cluster1.mgr.x.smithi078.stderr:2024-03-11T22:47:10.226+0000 7f1798442040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:47:10.328 INFO:tasks.ceph.cluster1.mgr.x.smithi078.stderr:2024-03-11T22:47:10.321+0000 7f1798442040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T22:47:10.482 INFO:tasks.ceph.cluster1.mgr.x.smithi078.stderr:2024-03-11T22:47:10.474+0000 7f1798442040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:47:10.523 INFO:tasks.ceph.cluster1.mgr.x.smithi078.stderr:2024-03-11T22:47:10.516+0000 7f1798442040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T22:47:10.677 INFO:tasks.ceph.cluster1.mgr.x.smithi078.stderr:2024-03-11T22:47:10.670+0000 7f1798442040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T22:47:11.797 INFO:tasks.ceph.cluster1.mgr.x.smithi078.stderr:2024-03-11T22:47:11.790+0000 7f1798442040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T22:47:12.100 INFO:tasks.ceph.cluster1.mgr.x.smithi078.stderr:2024-03-11T22:47:12.093+0000 7f1798442040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T22:47:12.152 INFO:tasks.ceph.cluster1.mgr.x.smithi078.stderr:2024-03-11T22:47:12.144+0000 7f1798442040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T22:47:12.269 INFO:tasks.ceph.cluster1.mgr.x.smithi078.stderr:2024-03-11T22:47:12.261+0000 7f1798442040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T22:47:12.738 INFO:tasks.ceph.cluster1.osd.1.smithi078.stderr:2024-03-11T22:47:12.731+0000 7f0d9a41b640 -1 osd.1 0 log_to_monitors true 2024-03-11T22:47:12.742 INFO:tasks.ceph.cluster1.osd.0.smithi078.stderr:2024-03-11T22:47:12.735+0000 7ff7bc010640 -1 osd.0 0 log_to_monitors true 2024-03-11T22:47:12.762 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2024-03-11T22:47:12.755+0000 7ff0d216d640 -1 osd.2 0 log_to_monitors true 2024-03-11T22:47:13.541 INFO:tasks.ceph.cluster1.mgr.x.smithi078.stderr:2024-03-11T22:47:13.534+0000 7f173989d640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-03-11T22:47:15.568 DEBUG:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-03-11T22:47:15.980 INFO:teuthology.misc.health.smithi078.stdout: 2024-03-11T22:47:15.981 INFO:teuthology.misc.health.smithi078.stdout:{"epoch":7,"fsid":"b6063557-edc9-4ae4-8432-61b8810d0c16","created":"2024-03-11T22:47:07.655058+0000","modified":"2024-03-11T22:47:14.525432+0000","last_up_change":"0.000000","last_in_change":"2024-03-11T22:47:08.512853+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"916b2b84-4c20-43d3-870e-890cfc5a42d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"44230f7e-eaff-4b71-b1cc-867151169e60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"42db0bd8-b0b4-40e6-aa12-ecffa1c99359","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T22:47:15.996 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-03-11T22:47:16.428 INFO:tasks.ceph.cluster1.osd.1.smithi078.stderr:2024-03-11T22:47:16.421+0000 7f0d954ca640 -1 osd.1 0 waiting for initial osdmap 2024-03-11T22:47:16.434 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2024-03-11T22:47:16.426+0000 7ff0cd8e0640 -1 osd.2 0 waiting for initial osdmap 2024-03-11T22:47:16.449 INFO:tasks.ceph.cluster1.osd.1.smithi078.stderr:2024-03-11T22:47:16.442+0000 7f0d8dafc640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:47:16.449 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2024-03-11T22:47:16.442+0000 7ff0c5f12640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:47:16.471 INFO:tasks.ceph.cluster1.osd.0.smithi078.stderr:2024-03-11T22:47:16.464+0000 7ff7b7252640 -1 osd.0 0 waiting for initial osdmap 2024-03-11T22:47:16.488 INFO:tasks.ceph.cluster1.osd.0.smithi078.stderr:2024-03-11T22:47:16.481+0000 7ff7af884640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:47:22.298 DEBUG:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-03-11T22:47:22.580 INFO:teuthology.misc.health.smithi078.stdout: 2024-03-11T22:47:22.581 INFO:teuthology.misc.health.smithi078.stdout:{"epoch":11,"fsid":"b6063557-edc9-4ae4-8432-61b8810d0c16","created":"2024-03-11T22:47:07.655058+0000","modified":"2024-03-11T22:47:20.538849+0000","last_up_change":"2024-03-11T22:47:17.431296+0000","last_in_change":"2024-03-11T22:47:08.512853+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T22:47:18.520646+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"916b2b84-4c20-43d3-870e-890cfc5a42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":611835560},{"type":"v1","addr":"172.21.15.78:6809","nonce":611835560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":611835560},{"type":"v1","addr":"172.21.15.78:6811","nonce":611835560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":611835560},{"type":"v1","addr":"172.21.15.78:6815","nonce":611835560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":611835560},{"type":"v1","addr":"172.21.15.78:6813","nonce":611835560}]},"public_addr":"172.21.15.78:6809/611835560","cluster_addr":"172.21.15.78:6811/611835560","heartbeat_back_addr":"172.21.15.78:6815/611835560","heartbeat_front_addr":"172.21.15.78:6813/611835560","state":["exists","up"]},{"osd":1,"uuid":"44230f7e-eaff-4b71-b1cc-867151169e60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3208301419},{"type":"v1","addr":"172.21.15.78:6801","nonce":3208301419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3208301419},{"type":"v1","addr":"172.21.15.78:6803","nonce":3208301419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3208301419},{"type":"v1","addr":"172.21.15.78:6807","nonce":3208301419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3208301419},{"type":"v1","addr":"172.21.15.78:6805","nonce":3208301419}]},"public_addr":"172.21.15.78:6801/3208301419","cluster_addr":"172.21.15.78:6803/3208301419","heartbeat_back_addr":"172.21.15.78:6807/3208301419","heartbeat_front_addr":"172.21.15.78:6805/3208301419","state":["exists","up"]},{"osd":2,"uuid":"42db0bd8-b0b4-40e6-aa12-ecffa1c99359","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3775303485},{"type":"v1","addr":"172.21.15.78:6817","nonce":3775303485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3775303485},{"type":"v1","addr":"172.21.15.78:6819","nonce":3775303485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3775303485},{"type":"v1","addr":"172.21.15.78:6823","nonce":3775303485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3775303485},{"type":"v1","addr":"172.21.15.78:6821","nonce":3775303485}]},"public_addr":"172.21.15.78:6817/3775303485","cluster_addr":"172.21.15.78:6819/3775303485","heartbeat_back_addr":"172.21.15.78:6823/3775303485","heartbeat_front_addr":"172.21.15.78:6821/3775303485","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:47:13.733519+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:47:13.712919+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:47:13.745240+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T22:47:22.592 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-03-11T22:47:22.592 INFO:tasks.ceph:Creating RBD pool 2024-03-11T22:47:22.592 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --cluster cluster1 osd pool create rbd 8 2024-03-11T22:47:23.553 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'rbd' created 2024-03-11T22:47:23.571 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --cluster cluster1 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-03-11T22:47:24.557 INFO:teuthology.orchestra.run.smithi078.stderr:enabled application 'rbd' on pool 'rbd' 2024-03-11T22:47:24.575 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2024-03-11T22:47:24.576 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2024-03-11T22:47:24.576 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2024-03-11T22:47:24.576 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2024-03-11T22:47:24.576 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-11T22:47:24.911 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:47:24.923 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":5,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":1005807340},{"type":"v1","addr":"172.21.15.78:6825","nonce":1005807340}]},"active_addr":"172.21.15.78:6825/1005807340","active_change":"2024-03-11T22:47:12.275018+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":246206263}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":2268994424}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":3058222159}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":21057031}]}]} 2024-03-11T22:47:24.925 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2024-03-11T22:47:24.925 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2024-03-11T22:47:24.925 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-03-11T22:47:25.206 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:47:25.206 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":13,"fsid":"b6063557-edc9-4ae4-8432-61b8810d0c16","created":"2024-03-11T22:47:07.655058+0000","modified":"2024-03-11T22:47:24.548018+0000","last_up_change":"2024-03-11T22:47:17.431296+0000","last_in_change":"2024-03-11T22:47:08.512853+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T22:47:18.520646+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-11T22:47:22.886451+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"916b2b84-4c20-43d3-870e-890cfc5a42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":611835560},{"type":"v1","addr":"172.21.15.78:6809","nonce":611835560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":611835560},{"type":"v1","addr":"172.21.15.78:6811","nonce":611835560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":611835560},{"type":"v1","addr":"172.21.15.78:6815","nonce":611835560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":611835560},{"type":"v1","addr":"172.21.15.78:6813","nonce":611835560}]},"public_addr":"172.21.15.78:6809/611835560","cluster_addr":"172.21.15.78:6811/611835560","heartbeat_back_addr":"172.21.15.78:6815/611835560","heartbeat_front_addr":"172.21.15.78:6813/611835560","state":["exists","up"]},{"osd":1,"uuid":"44230f7e-eaff-4b71-b1cc-867151169e60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3208301419},{"type":"v1","addr":"172.21.15.78:6801","nonce":3208301419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3208301419},{"type":"v1","addr":"172.21.15.78:6803","nonce":3208301419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3208301419},{"type":"v1","addr":"172.21.15.78:6807","nonce":3208301419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3208301419},{"type":"v1","addr":"172.21.15.78:6805","nonce":3208301419}]},"public_addr":"172.21.15.78:6801/3208301419","cluster_addr":"172.21.15.78:6803/3208301419","heartbeat_back_addr":"172.21.15.78:6807/3208301419","heartbeat_front_addr":"172.21.15.78:6805/3208301419","state":["exists","up"]},{"osd":2,"uuid":"42db0bd8-b0b4-40e6-aa12-ecffa1c99359","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3775303485},{"type":"v1","addr":"172.21.15.78:6817","nonce":3775303485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3775303485},{"type":"v1","addr":"172.21.15.78:6819","nonce":3775303485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3775303485},{"type":"v1","addr":"172.21.15.78:6823","nonce":3775303485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3775303485},{"type":"v1","addr":"172.21.15.78:6821","nonce":3775303485}]},"public_addr":"172.21.15.78:6817/3775303485","cluster_addr":"172.21.15.78:6819/3775303485","heartbeat_back_addr":"172.21.15.78:6823/3775303485","heartbeat_front_addr":"172.21.15.78:6821/3775303485","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:47:13.733519+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:47:13.712919+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:47:13.745240+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T22:47:25.219 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2024-03-11T22:47:25.219 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-03-11T22:47:25.500 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:47:25.501 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":13,"fsid":"b6063557-edc9-4ae4-8432-61b8810d0c16","created":"2024-03-11T22:47:07.655058+0000","modified":"2024-03-11T22:47:24.548018+0000","last_up_change":"2024-03-11T22:47:17.431296+0000","last_in_change":"2024-03-11T22:47:08.512853+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T22:47:18.520646+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-11T22:47:22.886451+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"916b2b84-4c20-43d3-870e-890cfc5a42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":611835560},{"type":"v1","addr":"172.21.15.78:6809","nonce":611835560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":611835560},{"type":"v1","addr":"172.21.15.78:6811","nonce":611835560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":611835560},{"type":"v1","addr":"172.21.15.78:6815","nonce":611835560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":611835560},{"type":"v1","addr":"172.21.15.78:6813","nonce":611835560}]},"public_addr":"172.21.15.78:6809/611835560","cluster_addr":"172.21.15.78:6811/611835560","heartbeat_back_addr":"172.21.15.78:6815/611835560","heartbeat_front_addr":"172.21.15.78:6813/611835560","state":["exists","up"]},{"osd":1,"uuid":"44230f7e-eaff-4b71-b1cc-867151169e60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3208301419},{"type":"v1","addr":"172.21.15.78:6801","nonce":3208301419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3208301419},{"type":"v1","addr":"172.21.15.78:6803","nonce":3208301419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3208301419},{"type":"v1","addr":"172.21.15.78:6807","nonce":3208301419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3208301419},{"type":"v1","addr":"172.21.15.78:6805","nonce":3208301419}]},"public_addr":"172.21.15.78:6801/3208301419","cluster_addr":"172.21.15.78:6803/3208301419","heartbeat_back_addr":"172.21.15.78:6807/3208301419","heartbeat_front_addr":"172.21.15.78:6805/3208301419","state":["exists","up"]},{"osd":2,"uuid":"42db0bd8-b0b4-40e6-aa12-ecffa1c99359","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3775303485},{"type":"v1","addr":"172.21.15.78:6817","nonce":3775303485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3775303485},{"type":"v1","addr":"172.21.15.78:6819","nonce":3775303485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3775303485},{"type":"v1","addr":"172.21.15.78:6823","nonce":3775303485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3775303485},{"type":"v1","addr":"172.21.15.78:6821","nonce":3775303485}]},"public_addr":"172.21.15.78:6817/3775303485","cluster_addr":"172.21.15.78:6819/3775303485","heartbeat_back_addr":"172.21.15.78:6823/3775303485","heartbeat_front_addr":"172.21.15.78:6821/3775303485","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:47:13.733519+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:47:13.712919+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:47:13.745240+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T22:47:25.513 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2024-03-11T22:47:25.514 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2024-03-11T22:47:25.514 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2024-03-11T22:47:25.657 INFO:teuthology.orchestra.run.smithi078.stdout:34359738374 2024-03-11T22:47:25.658 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-03-11T22:47:25.658 INFO:teuthology.orchestra.run.smithi078.stdout:34359738374 2024-03-11T22:47:25.659 INFO:teuthology.orchestra.run.smithi078.stdout:34359738373 2024-03-11T22:47:25.659 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-03-11T22:47:25.660 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-03-11T22:47:25.996 INFO:teuthology.orchestra.run.smithi078.stdout:34359738373 2024-03-11T22:47:25.999 INFO:teuthology.orchestra.run.smithi078.stdout:34359738373 2024-03-11T22:47:26.006 INFO:teuthology.orchestra.run.smithi078.stdout:34359738372 2024-03-11T22:47:26.010 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738373 for osd.0 2024-03-11T22:47:26.013 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738373 for osd.1 2024-03-11T22:47:26.019 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.2 2024-03-11T22:47:27.011 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-03-11T22:47:27.013 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-03-11T22:47:27.020 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-03-11T22:47:27.324 INFO:teuthology.orchestra.run.smithi078.stdout:34359738374 2024-03-11T22:47:27.338 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738374 for osd.1 2024-03-11T22:47:27.338 DEBUG:teuthology.parallel:result is None 2024-03-11T22:47:27.355 INFO:teuthology.orchestra.run.smithi078.stdout:34359738374 2024-03-11T22:47:27.368 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738374 for osd.0 2024-03-11T22:47:27.368 DEBUG:teuthology.parallel:result is None 2024-03-11T22:47:27.387 INFO:teuthology.orchestra.run.smithi078.stdout:34359738373 2024-03-11T22:47:27.400 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.2 2024-03-11T22:47:27.400 DEBUG:teuthology.parallel:result is None 2024-03-11T22:47:27.400 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-03-11T22:47:27.401 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-03-11T22:47:27.722 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:47:27.722 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-03-11T22:47:27.734 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-03-11T22:47:26.280727+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82560,"kb_used_data":1496,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032960,"statfs":{"total":289910292480,"available":289825751040,"internally_reserved":0,"allocated":1531904,"data_stored":1274801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.732311"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-11T22:47:25.560302+0000","last_change":"2024-03-11T22:47:24.560792+0000","last_active":"2024-03-11T22:47:25.560302+0000","last_peered":"2024-03-11T22:47:25.560302+0000","last_clean":"2024-03-11T22:47:25.560302+0000","last_became_active":"2024-03-11T22:47:24.560512+0000","last_became_peered":"2024-03-11T22:47:24.560512+0000","last_unstale":"2024-03-11T22:47:25.560302+0000","last_undegraded":"2024-03-11T22:47:25.560302+0000","last_fullsized":"2024-03-11T22:47:25.560302+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:47:23.543557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:47:23.543557+0000","last_clean_scrub_stamp":"2024-03-11T22:47:23.543557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:01:37.184425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-11T22:47:25.560462+0000","last_change":"2024-03-11T22:47:24.558647+0000","last_active":"2024-03-11T22:47:25.560462+0000","last_peered":"2024-03-11T22:47:25.560462+0000","last_clean":"2024-03-11T22:47:25.560462+0000","last_became_active":"2024-03-11T22:47:24.558372+0000","last_became_peered":"2024-03-11T22:47:24.558372+0000","last_unstale":"2024-03-11T22:47:25.560462+0000","last_undegraded":"2024-03-11T22:47:25.560462+0000","last_fullsized":"2024-03-11T22:47:25.560462+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:47:23.543557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:47:23.543557+0000","last_clean_scrub_stamp":"2024-03-11T22:47:23.543557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:46:06.979393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-11T22:47:25.560624+0000","last_change":"2024-03-11T22:47:24.561087+0000","last_active":"2024-03-11T22:47:25.560624+0000","last_peered":"2024-03-11T22:47:25.560624+0000","last_clean":"2024-03-11T22:47:25.560624+0000","last_became_active":"2024-03-11T22:47:24.560525+0000","last_became_peered":"2024-03-11T22:47:24.560525+0000","last_unstale":"2024-03-11T22:47:25.560624+0000","last_undegraded":"2024-03-11T22:47:25.560624+0000","last_fullsized":"2024-03-11T22:47:25.560624+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:47:23.543557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:47:23.543557+0000","last_clean_scrub_stamp":"2024-03-11T22:47:23.543557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:23:07.990764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-11T22:47:25.560259+0000","last_change":"2024-03-11T22:47:24.560024+0000","last_active":"2024-03-11T22:47:25.560259+0000","last_peered":"2024-03-11T22:47:25.560259+0000","last_clean":"2024-03-11T22:47:25.560259+0000","last_became_active":"2024-03-11T22:47:24.559746+0000","last_became_peered":"2024-03-11T22:47:24.559746+0000","last_unstale":"2024-03-11T22:47:25.560259+0000","last_undegraded":"2024-03-11T22:47:25.560259+0000","last_fullsized":"2024-03-11T22:47:25.560259+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:47:23.543557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:47:23.543557+0000","last_clean_scrub_stamp":"2024-03-11T22:47:23.543557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:14:47.379366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-11T22:47:25.560460+0000","last_change":"2024-03-11T22:47:24.558572+0000","last_active":"2024-03-11T22:47:25.560460+0000","last_peered":"2024-03-11T22:47:25.560460+0000","last_clean":"2024-03-11T22:47:25.560460+0000","last_became_active":"2024-03-11T22:47:24.558275+0000","last_became_peered":"2024-03-11T22:47:24.558275+0000","last_unstale":"2024-03-11T22:47:25.560460+0000","last_undegraded":"2024-03-11T22:47:25.560460+0000","last_fullsized":"2024-03-11T22:47:25.560460+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:47:23.543557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:47:23.543557+0000","last_clean_scrub_stamp":"2024-03-11T22:47:23.543557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:57:49.041690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-11T22:47:25.555794+0000","last_change":"2024-03-11T22:47:24.559710+0000","last_active":"2024-03-11T22:47:25.555794+0000","last_peered":"2024-03-11T22:47:25.555794+0000","last_clean":"2024-03-11T22:47:25.555794+0000","last_became_active":"2024-03-11T22:47:24.559425+0000","last_became_peered":"2024-03-11T22:47:24.559425+0000","last_unstale":"2024-03-11T22:47:25.555794+0000","last_undegraded":"2024-03-11T22:47:25.555794+0000","last_fullsized":"2024-03-11T22:47:25.555794+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:47:23.543557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:47:23.543557+0000","last_clean_scrub_stamp":"2024-03-11T22:47:23.543557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:48:17.331990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-11T22:47:25.555678+0000","last_change":"2024-03-11T22:47:24.559960+0000","last_active":"2024-03-11T22:47:25.555678+0000","last_peered":"2024-03-11T22:47:25.555678+0000","last_clean":"2024-03-11T22:47:25.555678+0000","last_became_active":"2024-03-11T22:47:24.559667+0000","last_became_peered":"2024-03-11T22:47:24.559667+0000","last_unstale":"2024-03-11T22:47:25.555678+0000","last_undegraded":"2024-03-11T22:47:25.555678+0000","last_fullsized":"2024-03-11T22:47:25.555678+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:47:23.543557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:47:23.543557+0000","last_clean_scrub_stamp":"2024-03-11T22:47:23.543557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:20:05.119812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-11T22:47:25.560334+0000","last_change":"2024-03-11T22:47:24.559307+0000","last_active":"2024-03-11T22:47:25.560334+0000","last_peered":"2024-03-11T22:47:25.560334+0000","last_clean":"2024-03-11T22:47:25.560334+0000","last_became_active":"2024-03-11T22:47:24.559024+0000","last_became_peered":"2024-03-11T22:47:24.559024+0000","last_unstale":"2024-03-11T22:47:25.560334+0000","last_undegraded":"2024-03-11T22:47:25.560334+0000","last_fullsized":"2024-03-11T22:47:25.560334+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:47:23.543557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:47:23.543557+0000","last_clean_scrub_stamp":"2024-03-11T22:47:23.543557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:11:40.329267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":145,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-11T22:47:25.560200+0000","last_change":"2024-03-11T22:47:19.816641+0000","last_active":"2024-03-11T22:47:25.560200+0000","last_peered":"2024-03-11T22:47:25.560200+0000","last_clean":"2024-03-11T22:47:25.560200+0000","last_became_active":"2024-03-11T22:47:19.815500+0000","last_became_peered":"2024-03-11T22:47:19.815500+0000","last_unstale":"2024-03-11T22:47:25.560200+0000","last_undegraded":"2024-03-11T22:47:25.560200+0000","last_fullsized":"2024-03-11T22:47:25.560200+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:47:18.536874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:47:18.536874+0000","last_clean_scrub_stamp":"2024-03-11T22:47:18.536874+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:07:26.490231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27136,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":114688,"data_stored":31355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:47:27.735 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-03-11T22:47:28.013 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:47:28.013 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-03-11T22:47:28.025 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-03-11T22:47:26.280727+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82560,"kb_used_data":1496,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032960,"statfs":{"total":289910292480,"available":289825751040,"internally_reserved":0,"allocated":1531904,"data_stored":1274801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.732311"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-11T22:47:25.560302+0000","last_change":"2024-03-11T22:47:24.560792+0000","last_active":"2024-03-11T22:47:25.560302+0000","last_peered":"2024-03-11T22:47:25.560302+0000","last_clean":"2024-03-11T22:47:25.560302+0000","last_became_active":"2024-03-11T22:47:24.560512+0000","last_became_peered":"2024-03-11T22:47:24.560512+0000","last_unstale":"2024-03-11T22:47:25.560302+0000","last_undegraded":"2024-03-11T22:47:25.560302+0000","last_fullsized":"2024-03-11T22:47:25.560302+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:47:23.543557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:47:23.543557+0000","last_clean_scrub_stamp":"2024-03-11T22:47:23.543557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:01:37.184425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-11T22:47:25.560462+0000","last_change":"2024-03-11T22:47:24.558647+0000","last_active":"2024-03-11T22:47:25.560462+0000","last_peered":"2024-03-11T22:47:25.560462+0000","last_clean":"2024-03-11T22:47:25.560462+0000","last_became_active":"2024-03-11T22:47:24.558372+0000","last_became_peered":"2024-03-11T22:47:24.558372+0000","last_unstale":"2024-03-11T22:47:25.560462+0000","last_undegraded":"2024-03-11T22:47:25.560462+0000","last_fullsized":"2024-03-11T22:47:25.560462+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:47:23.543557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:47:23.543557+0000","last_clean_scrub_stamp":"2024-03-11T22:47:23.543557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:46:06.979393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-11T22:47:25.560624+0000","last_change":"2024-03-11T22:47:24.561087+0000","last_active":"2024-03-11T22:47:25.560624+0000","last_peered":"2024-03-11T22:47:25.560624+0000","last_clean":"2024-03-11T22:47:25.560624+0000","last_became_active":"2024-03-11T22:47:24.560525+0000","last_became_peered":"2024-03-11T22:47:24.560525+0000","last_unstale":"2024-03-11T22:47:25.560624+0000","last_undegraded":"2024-03-11T22:47:25.560624+0000","last_fullsized":"2024-03-11T22:47:25.560624+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:47:23.543557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:47:23.543557+0000","last_clean_scrub_stamp":"2024-03-11T22:47:23.543557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:23:07.990764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-11T22:47:25.560259+0000","last_change":"2024-03-11T22:47:24.560024+0000","last_active":"2024-03-11T22:47:25.560259+0000","last_peered":"2024-03-11T22:47:25.560259+0000","last_clean":"2024-03-11T22:47:25.560259+0000","last_became_active":"2024-03-11T22:47:24.559746+0000","last_became_peered":"2024-03-11T22:47:24.559746+0000","last_unstale":"2024-03-11T22:47:25.560259+0000","last_undegraded":"2024-03-11T22:47:25.560259+0000","last_fullsized":"2024-03-11T22:47:25.560259+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:47:23.543557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:47:23.543557+0000","last_clean_scrub_stamp":"2024-03-11T22:47:23.543557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:14:47.379366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-11T22:47:25.560460+0000","last_change":"2024-03-11T22:47:24.558572+0000","last_active":"2024-03-11T22:47:25.560460+0000","last_peered":"2024-03-11T22:47:25.560460+0000","last_clean":"2024-03-11T22:47:25.560460+0000","last_became_active":"2024-03-11T22:47:24.558275+0000","last_became_peered":"2024-03-11T22:47:24.558275+0000","last_unstale":"2024-03-11T22:47:25.560460+0000","last_undegraded":"2024-03-11T22:47:25.560460+0000","last_fullsized":"2024-03-11T22:47:25.560460+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:47:23.543557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:47:23.543557+0000","last_clean_scrub_stamp":"2024-03-11T22:47:23.543557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:57:49.041690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-11T22:47:25.555794+0000","last_change":"2024-03-11T22:47:24.559710+0000","last_active":"2024-03-11T22:47:25.555794+0000","last_peered":"2024-03-11T22:47:25.555794+0000","last_clean":"2024-03-11T22:47:25.555794+0000","last_became_active":"2024-03-11T22:47:24.559425+0000","last_became_peered":"2024-03-11T22:47:24.559425+0000","last_unstale":"2024-03-11T22:47:25.555794+0000","last_undegraded":"2024-03-11T22:47:25.555794+0000","last_fullsized":"2024-03-11T22:47:25.555794+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:47:23.543557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:47:23.543557+0000","last_clean_scrub_stamp":"2024-03-11T22:47:23.543557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:48:17.331990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-11T22:47:25.555678+0000","last_change":"2024-03-11T22:47:24.559960+0000","last_active":"2024-03-11T22:47:25.555678+0000","last_peered":"2024-03-11T22:47:25.555678+0000","last_clean":"2024-03-11T22:47:25.555678+0000","last_became_active":"2024-03-11T22:47:24.559667+0000","last_became_peered":"2024-03-11T22:47:24.559667+0000","last_unstale":"2024-03-11T22:47:25.555678+0000","last_undegraded":"2024-03-11T22:47:25.555678+0000","last_fullsized":"2024-03-11T22:47:25.555678+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:47:23.543557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:47:23.543557+0000","last_clean_scrub_stamp":"2024-03-11T22:47:23.543557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:20:05.119812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-11T22:47:25.560334+0000","last_change":"2024-03-11T22:47:24.559307+0000","last_active":"2024-03-11T22:47:25.560334+0000","last_peered":"2024-03-11T22:47:25.560334+0000","last_clean":"2024-03-11T22:47:25.560334+0000","last_became_active":"2024-03-11T22:47:24.559024+0000","last_became_peered":"2024-03-11T22:47:24.559024+0000","last_unstale":"2024-03-11T22:47:25.560334+0000","last_undegraded":"2024-03-11T22:47:25.560334+0000","last_fullsized":"2024-03-11T22:47:25.560334+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:47:23.543557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:47:23.543557+0000","last_clean_scrub_stamp":"2024-03-11T22:47:23.543557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:11:40.329267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":145,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-11T22:47:25.560200+0000","last_change":"2024-03-11T22:47:19.816641+0000","last_active":"2024-03-11T22:47:25.560200+0000","last_peered":"2024-03-11T22:47:25.560200+0000","last_clean":"2024-03-11T22:47:25.560200+0000","last_became_active":"2024-03-11T22:47:19.815500+0000","last_became_peered":"2024-03-11T22:47:19.815500+0000","last_unstale":"2024-03-11T22:47:25.560200+0000","last_undegraded":"2024-03-11T22:47:25.560200+0000","last_fullsized":"2024-03-11T22:47:25.560200+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:47:18.536874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:47:18.536874+0000","last_clean_scrub_stamp":"2024-03-11T22:47:18.536874+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:07:26.490231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27136,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":114688,"data_stored":31355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:47:28.026 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-03-11T22:47:28.026 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2024-03-11T22:47:28.026 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2024-03-11T22:47:28.026 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2024-03-11T22:47:28.342 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:47:28.343 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-11T22:47:28.355 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2024-03-11T22:47:28.355 INFO:teuthology.run_tasks:Running task ceph... 2024-03-11T22:47:28.366 INFO:tasks.ceph:Creating ceph cluster cluster2... 2024-03-11T22:47:28.366 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-11T22:47:28.367 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dis-2024-03-11_22:26:48-rbd-wip-dis-testing-distro-default-smithi/7592296', 'branch': 'wip-dis-testing', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum}', 'email': 'idryomov@gmail.com', 'first_in_suite': False, 'job_id': '7592296', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'dis-2024-03-11_22:26:48-rbd-wip-dis-testing-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-dis-testing'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'b117a47aa0ad81d207fdd9ba852068f2f262fea8'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'b117a47aa0ad81d207fdd9ba852068f2f262fea8'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-dis-testing-qa', 'sha1': 'a4679433626e24f222f64f757c6ab31760167603'}}, 'owner': 'scheduled_dis@teuthology', 'priority': 70, 'repo': 'https://github.com/ceph/ceph-ci', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 5633, 'sha1': 'b117a47aa0ad81d207fdd9ba852068f2f262fea8', 'sleep_before_teardown': 0, 'subset': '81/99', 'suite': 'rbd', 'suite_branch': 'wip-dis-testing-qa', 'suite_path': '/home/teuthworker/src/github.com_idryomov_ceph_a4679433626e24f222f64f757c6ab31760167603/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/idryomov/ceph', 'suite_sha1': 'a4679433626e24f222f64f757c6ab31760167603', 'targets': {'smithi078.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBALYIufoA3dz1qKE8kADQ7x1+cDKs+cZ3+eoCp4Ma9/PkJRs0d6nG11XqINqKSmIeLmh1eTH34m/OIJCkaVKVzE=', 'smithi132.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPNf1BzILs3QC7X/vb7hhud5sQ79Ov/m2HPKnz8nSwAUei2BPhVJ2BlnaQ0g4BlXne1kwcu0uHG9hKemdzGeUQw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'b117a47aa0ad81d207fdd9ba852068f2f262fea8'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'b117a47aa0ad81d207fdd9ba852068f2f262fea8'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'b117a47aa0ad81d207fdd9ba852068f2f262fea8'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-11_22:26:48', 'tube': 'smithi', 'user': 'dis', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-11T22:47:28.367 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-03-11T22:47:28.369 DEBUG:teuthology.orchestra.run.smithi132:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-03-11T22:47:28.387 DEBUG:teuthology.orchestra.run.smithi078:> sudo install -d -m0777 -- /var/run/ceph 2024-03-11T22:47:28.424 DEBUG:teuthology.orchestra.run.smithi132:> sudo install -d -m0777 -- /var/run/ceph 2024-03-11T22:47:28.458 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:47:28.458 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T22:47:28.515 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T22:47:28.515 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_1 2024-03-11T22:47:28.571 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T22:47:28.571 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:47:28.571 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-11T22:47:28.571 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:47:28.571 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:47:28.571 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-03-11 22:46:45.951509409 +0000 2024-03-11T22:47:28.571 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-03-11 22:43:56.653621153 +0000 2024-03-11T22:47:28.571 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-03-11 22:43:56.653621153 +0000 2024-03-11T22:47:28.572 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2024-03-11 22:43:56.653621153 +0000 2024-03-11T22:47:28.572 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T22:47:28.638 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-03-11T22:47:28.639 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-03-11T22:47:28.639 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000215792 s, 2.4 MB/s 2024-03-11T22:47:28.641 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T22:47:28.698 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_2 2024-03-11T22:47:28.754 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T22:47:28.754 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:47:28.754 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-11T22:47:28.754 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:47:28.754 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:47:28.754 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-03-11 22:46:45.951509409 +0000 2024-03-11T22:47:28.754 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-03-11 22:43:56.995612273 +0000 2024-03-11T22:47:28.754 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-03-11 22:43:56.995612273 +0000 2024-03-11T22:47:28.754 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2024-03-11 22:43:56.995612273 +0000 2024-03-11T22:47:28.755 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T22:47:28.820 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-03-11T22:47:28.820 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-03-11T22:47:28.820 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000201132 s, 2.5 MB/s 2024-03-11T22:47:28.822 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T22:47:28.878 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_3 2024-03-11T22:47:28.934 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T22:47:28.934 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:47:28.934 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-11T22:47:28.934 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:47:28.934 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:47:28.934 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-03-11 22:46:45.952509383 +0000 2024-03-11T22:47:28.934 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-03-11 22:43:57.329603601 +0000 2024-03-11T22:47:28.934 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-03-11 22:43:57.329603601 +0000 2024-03-11T22:47:28.934 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2024-03-11 22:43:57.329603601 +0000 2024-03-11T22:47:28.935 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T22:47:29.000 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-03-11T22:47:29.000 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-03-11T22:47:29.000 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000189439 s, 2.7 MB/s 2024-03-11T22:47:29.001 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T22:47:29.058 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_4 2024-03-11T22:47:29.113 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T22:47:29.113 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:47:29.113 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-11T22:47:29.113 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:47:29.113 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:47:29.113 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-03-11 22:46:45.952509383 +0000 2024-03-11T22:47:29.113 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-03-11 22:43:57.675594617 +0000 2024-03-11T22:47:29.113 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-03-11 22:43:57.675594617 +0000 2024-03-11T22:47:29.113 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2024-03-11 22:43:57.675594617 +0000 2024-03-11T22:47:29.114 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T22:47:29.180 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-03-11T22:47:29.180 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-03-11T22:47:29.180 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000239108 s, 2.1 MB/s 2024-03-11T22:47:29.181 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T22:47:29.237 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-03-11T22:47:29.237 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi132.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-03-11T22:47:29.238 INFO:tasks.ceph:Generating config... 2024-03-11T22:47:29.239 INFO:tasks.ceph:[client] debug rbd = 20 2024-03-11T22:47:29.239 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-03-11T22:47:29.239 INFO:tasks.ceph:[client] log to stderr = False 2024-03-11T22:47:29.239 INFO:tasks.ceph:[client] rbd default features = 125 2024-03-11T22:47:29.239 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2024-03-11T22:47:29.239 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-11T22:47:29.240 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-11T22:47:29.240 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-11T22:47:29.240 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-11T22:47:29.240 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-11T22:47:29.240 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-11T22:47:29.240 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-11T22:47:29.240 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-11T22:47:29.240 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-11T22:47:29.240 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-03-11T22:47:29.240 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-11T22:47:29.240 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-11T22:47:29.240 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-11T22:47:29.240 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-11T22:47:29.240 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-11T22:47:29.240 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-11T22:47:29.241 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-11T22:47:29.241 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-11T22:47:29.241 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-11T22:47:29.241 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-11T22:47:29.241 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-11T22:47:29.241 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-03-11T22:47:29.241 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-03-11T22:47:29.241 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-11T22:47:29.241 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-11T22:47:29.241 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-03-11T22:47:29.241 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-11T22:47:29.241 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-11T22:47:29.241 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-11T22:47:29.241 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-11T22:47:29.241 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-11T22:47:29.242 INFO:tasks.ceph:Setting up cluster2.mon.a... 2024-03-11T22:47:29.242 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2024-03-11T22:47:29.319 INFO:teuthology.orchestra.run.smithi132.stdout:creating /etc/ceph/cluster2.keyring 2024-03-11T22:47:29.321 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2024-03-11T22:47:29.406 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-03-11T22:47:29.433 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.132')] 2024-03-11T22:47:29.433 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.132', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-03-11T22:47:29.434 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:47:29.434 DEBUG:teuthology.orchestra.run.smithi132:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-11T22:47:29.488 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.132 --print /home/ubuntu/cephtest/cluster2.monmap 2024-03-11T22:47:29.567 INFO:teuthology.orchestra.run.smithi132.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2024-03-11T22:47:29.568 INFO:teuthology.orchestra.run.smithi132.stdout:monmaptool: generated fsid f1daf4d2-a7c6-44cf-bbbf-27e068deb999 2024-03-11T22:47:29.568 INFO:teuthology.orchestra.run.smithi132.stdout:setting min_mon_release = quincy 2024-03-11T22:47:29.568 INFO:teuthology.orchestra.run.smithi132.stdout:epoch 0 2024-03-11T22:47:29.568 INFO:teuthology.orchestra.run.smithi132.stdout:fsid f1daf4d2-a7c6-44cf-bbbf-27e068deb999 2024-03-11T22:47:29.568 INFO:teuthology.orchestra.run.smithi132.stdout:last_changed 2024-03-11T22:47:29.561726+0000 2024-03-11T22:47:29.568 INFO:teuthology.orchestra.run.smithi132.stdout:created 2024-03-11T22:47:29.561726+0000 2024-03-11T22:47:29.568 INFO:teuthology.orchestra.run.smithi132.stdout:min_mon_release 17 (quincy) 2024-03-11T22:47:29.568 INFO:teuthology.orchestra.run.smithi132.stdout:election_strategy: 1 2024-03-11T22:47:29.568 INFO:teuthology.orchestra.run.smithi132.stdout:0: [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon.a 2024-03-11T22:47:29.568 INFO:teuthology.orchestra.run.smithi132.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2024-03-11T22:47:29.569 DEBUG:teuthology.orchestra.run.smithi132:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-11T22:47:29.624 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID f1daf4d2-a7c6-44cf-bbbf-27e068deb999... 2024-03-11T22:47:29.625 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-03-11T22:47:29.628 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-03-11T22:47:29.694 INFO:teuthology.orchestra.run.smithi078.stdout:[global] 2024-03-11T22:47:29.694 INFO:teuthology.orchestra.run.smithi078.stdout: chdir = "" 2024-03-11T22:47:29.695 INFO:teuthology.orchestra.run.smithi078.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-11T22:47:29.695 INFO:teuthology.orchestra.run.smithi078.stdout: auth supported = cephx 2024-03-11T22:47:29.695 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:47:29.695 INFO:teuthology.orchestra.run.smithi078.stdout: filestore xattr use omap = true 2024-03-11T22:47:29.695 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:47:29.695 INFO:teuthology.orchestra.run.smithi078.stdout: mon clock drift allowed = 1.000 2024-03-11T22:47:29.695 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:47:29.695 INFO:teuthology.orchestra.run.smithi078.stdout: osd crush chooseleaf type = 0 2024-03-11T22:47:29.695 INFO:teuthology.orchestra.run.smithi078.stdout: auth debug = true 2024-03-11T22:47:29.695 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:47:29.695 INFO:teuthology.orchestra.run.smithi078.stdout: ms die on old message = true 2024-03-11T22:47:29.695 INFO:teuthology.orchestra.run.smithi078.stdout: ms die on bug = true 2024-03-11T22:47:29.695 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:47:29.695 INFO:teuthology.orchestra.run.smithi078.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-11T22:47:29.695 INFO:teuthology.orchestra.run.smithi078.stdout: mon pg warn max object skew = 0 2024-03-11T22:47:29.696 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:47:29.696 INFO:teuthology.orchestra.run.smithi078.stdout: # disable pg_autoscaler by default for new pools 2024-03-11T22:47:29.696 INFO:teuthology.orchestra.run.smithi078.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-11T22:47:29.696 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:47:29.696 INFO:teuthology.orchestra.run.smithi078.stdout: osd pool default size = 2 2024-03-11T22:47:29.696 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:47:29.696 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd allow primary affinity = true 2024-03-11T22:47:29.696 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd allow pg remap = true 2024-03-11T22:47:29.696 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on legacy crush tunables = false 2024-03-11T22:47:29.696 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on crush straw calc version zero = false 2024-03-11T22:47:29.696 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on no sortbitwise = false 2024-03-11T22:47:29.696 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on osd down out interval zero = false 2024-03-11T22:47:29.696 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on too few osds = false 2024-03-11T22:47:29.696 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-11T22:47:29.696 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-11T22:47:29.697 INFO:teuthology.orchestra.run.smithi078.stdout: mon_allow_pool_size_one = true 2024-03-11T22:47:29.697 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:47:29.697 INFO:teuthology.orchestra.run.smithi078.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-11T22:47:29.697 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:47:29.697 INFO:teuthology.orchestra.run.smithi078.stdout: osd default data pool replay window = 5 2024-03-11T22:47:29.697 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:47:29.697 INFO:teuthology.orchestra.run.smithi078.stdout: mon allow pool delete = true 2024-03-11T22:47:29.697 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:47:29.697 INFO:teuthology.orchestra.run.smithi078.stdout: mon cluster log file level = debug 2024-03-11T22:47:29.697 INFO:teuthology.orchestra.run.smithi078.stdout: debug asserts on shutdown = true 2024-03-11T22:47:29.697 INFO:teuthology.orchestra.run.smithi078.stdout: mon health detail to clog = false 2024-03-11T22:47:29.697 INFO:teuthology.orchestra.run.smithi078.stdout: mon host = 172.21.15.132 2024-03-11T22:47:29.697 INFO:teuthology.orchestra.run.smithi078.stdout: mon client directed command retry = 5 2024-03-11T22:47:29.697 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on pool no app = False 2024-03-11T22:47:29.697 INFO:teuthology.orchestra.run.smithi078.stdout: ms inject socket failures = 5000 2024-03-11T22:47:29.697 INFO:teuthology.orchestra.run.smithi078.stdout: fsid = f1daf4d2-a7c6-44cf-bbbf-27e068deb999 2024-03-11T22:47:29.698 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:47:29.698 INFO:teuthology.orchestra.run.smithi078.stdout:[osd] 2024-03-11T22:47:29.698 INFO:teuthology.orchestra.run.smithi078.stdout: osd journal size = 100 2024-03-11T22:47:29.698 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:47:29.698 INFO:teuthology.orchestra.run.smithi078.stdout: osd scrub load threshold = 5.0 2024-03-11T22:47:29.698 INFO:teuthology.orchestra.run.smithi078.stdout: osd scrub max interval = 600 2024-03-11T22:47:29.698 INFO:teuthology.orchestra.run.smithi078.stdout: osd mclock profile = high_recovery_ops 2024-03-11T22:47:29.698 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:47:29.698 INFO:teuthology.orchestra.run.smithi078.stdout: osd recover clone overlap = true 2024-03-11T22:47:29.698 INFO:teuthology.orchestra.run.smithi078.stdout: osd recovery max chunk = 1048576 2024-03-11T22:47:29.698 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:47:29.698 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug shutdown = true 2024-03-11T22:47:29.698 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug op order = true 2024-03-11T22:47:29.698 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug verify stray on activate = true 2024-03-11T22:47:29.699 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug trim objects = true 2024-03-11T22:47:29.699 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:47:29.699 INFO:teuthology.orchestra.run.smithi078.stdout: osd open classes on start = true 2024-03-11T22:47:29.699 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug pg log writeout = true 2024-03-11T22:47:29.699 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:47:29.699 INFO:teuthology.orchestra.run.smithi078.stdout: osd deep scrub update digest min age = 30 2024-03-11T22:47:29.699 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:47:29.699 INFO:teuthology.orchestra.run.smithi078.stdout: osd map max advance = 10 2024-03-11T22:47:29.699 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:47:29.699 INFO:teuthology.orchestra.run.smithi078.stdout: journal zero on create = true 2024-03-11T22:47:29.699 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:47:29.699 INFO:teuthology.orchestra.run.smithi078.stdout: filestore ondisk finisher threads = 3 2024-03-11T22:47:29.699 INFO:teuthology.orchestra.run.smithi078.stdout: filestore apply finisher threads = 3 2024-03-11T22:47:29.699 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:47:29.699 INFO:teuthology.orchestra.run.smithi078.stdout: bdev debug aio = true 2024-03-11T22:47:29.700 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug misdirected ops = true 2024-03-11T22:47:29.700 INFO:teuthology.orchestra.run.smithi078.stdout: bdev async discard = True 2024-03-11T22:47:29.700 INFO:teuthology.orchestra.run.smithi078.stdout: bdev enable discard = True 2024-03-11T22:47:29.700 INFO:teuthology.orchestra.run.smithi078.stdout: bluestore allocator = stupid 2024-03-11T22:47:29.700 INFO:teuthology.orchestra.run.smithi078.stdout: bluestore block size = 96636764160 2024-03-11T22:47:29.700 INFO:teuthology.orchestra.run.smithi078.stdout: bluestore fsck on mount = True 2024-03-11T22:47:29.700 INFO:teuthology.orchestra.run.smithi078.stdout: debug bluefs = 1/20 2024-03-11T22:47:29.700 INFO:teuthology.orchestra.run.smithi078.stdout: debug bluestore = 1/20 2024-03-11T22:47:29.700 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2024-03-11T22:47:29.700 INFO:teuthology.orchestra.run.smithi078.stdout: debug osd = 20 2024-03-11T22:47:29.700 INFO:teuthology.orchestra.run.smithi078.stdout: debug rocksdb = 4/10 2024-03-11T22:47:29.700 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd backfillfull_ratio = 0.85 2024-03-11T22:47:29.700 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd full ratio = 0.9 2024-03-11T22:47:29.700 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd nearfull ratio = 0.8 2024-03-11T22:47:29.700 INFO:teuthology.orchestra.run.smithi078.stdout: osd failsafe full ratio = 0.95 2024-03-11T22:47:29.701 INFO:teuthology.orchestra.run.smithi078.stdout: osd objectstore = bluestore 2024-03-11T22:47:29.701 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:47:29.701 INFO:teuthology.orchestra.run.smithi078.stdout:[mgr] 2024-03-11T22:47:29.701 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2024-03-11T22:47:29.701 INFO:teuthology.orchestra.run.smithi078.stdout: debug mgr = 20 2024-03-11T22:47:29.701 INFO:teuthology.orchestra.run.smithi078.stdout: debug mon = 20 2024-03-11T22:47:29.701 INFO:teuthology.orchestra.run.smithi078.stdout: debug auth = 20 2024-03-11T22:47:29.701 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min pgs per osd = 4 2024-03-11T22:47:29.701 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min bytes per osd = 10 2024-03-11T22:47:29.701 INFO:teuthology.orchestra.run.smithi078.stdout: mgr/telemetry/nag = false 2024-03-11T22:47:29.701 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:47:29.701 INFO:teuthology.orchestra.run.smithi078.stdout:[mon] 2024-03-11T22:47:29.701 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2024-03-11T22:47:29.702 INFO:teuthology.orchestra.run.smithi078.stdout: debug mon = 20 2024-03-11T22:47:29.702 INFO:teuthology.orchestra.run.smithi078.stdout: debug paxos = 20 2024-03-11T22:47:29.702 INFO:teuthology.orchestra.run.smithi078.stdout: debug auth = 20 2024-03-11T22:47:29.702 INFO:teuthology.orchestra.run.smithi078.stdout: mon data avail warn = 5 2024-03-11T22:47:29.702 INFO:teuthology.orchestra.run.smithi078.stdout: mon mgr mkfs grace = 240 2024-03-11T22:47:29.702 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min pgs per osd = 4 2024-03-11T22:47:29.702 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd reporter subtree level = osd 2024-03-11T22:47:29.702 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd prime pg temp = true 2024-03-11T22:47:29.702 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min bytes per osd = 10 2024-03-11T22:47:29.702 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:47:29.702 INFO:teuthology.orchestra.run.smithi078.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-11T22:47:29.702 INFO:teuthology.orchestra.run.smithi078.stdout: auth mon ticket ttl = 660 # 11m 2024-03-11T22:47:29.702 INFO:teuthology.orchestra.run.smithi078.stdout: auth service ticket ttl = 240 # 4m 2024-03-11T22:47:29.702 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:47:29.702 INFO:teuthology.orchestra.run.smithi078.stdout: # don't complain about insecure global_id in the test suite 2024-03-11T22:47:29.703 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-11T22:47:29.703 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-11T22:47:29.706 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:47:29.706 INFO:teuthology.orchestra.run.smithi078.stdout: # 1m isn't quite enough 2024-03-11T22:47:29.706 INFO:teuthology.orchestra.run.smithi078.stdout: mon_down_mkfs_grace = 2m 2024-03-11T22:47:29.706 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:47:29.707 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_filestore_osds = false 2024-03-11T22:47:29.707 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-11T22:47:29.707 INFO:teuthology.orchestra.run.smithi078.stdout:[client] 2024-03-11T22:47:29.707 INFO:teuthology.orchestra.run.smithi078.stdout: rgw cache enabled = true 2024-03-11T22:47:29.707 INFO:teuthology.orchestra.run.smithi078.stdout: rgw enable ops log = true 2024-03-11T22:47:29.707 INFO:teuthology.orchestra.run.smithi078.stdout: rgw enable usage log = true 2024-03-11T22:47:29.707 INFO:teuthology.orchestra.run.smithi078.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-11T22:47:29.707 INFO:teuthology.orchestra.run.smithi078.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-11T22:47:29.707 INFO:teuthology.orchestra.run.smithi078.stdout: debug rbd = 20 2024-03-11T22:47:29.707 INFO:teuthology.orchestra.run.smithi078.stdout: debug rbd_mirror = 15 2024-03-11T22:47:29.707 INFO:teuthology.orchestra.run.smithi078.stdout: log to stderr = False 2024-03-11T22:47:29.707 INFO:teuthology.orchestra.run.smithi078.stdout: rbd default features = 125 2024-03-11T22:47:29.707 INFO:teuthology.orchestra.run.smithi078.stdout: rbd mirror image policy type = simple 2024-03-11T22:47:29.707 INFO:teuthology.orchestra.run.smithi078.stdout:[mon.a] 2024-03-11T22:47:29.708 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror.0] 2024-03-11T22:47:29.708 INFO:teuthology.orchestra.run.smithi078.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-11T22:47:29.708 INFO:teuthology.orchestra.run.smithi078.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-11T22:47:29.708 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror.1] 2024-03-11T22:47:29.708 INFO:teuthology.orchestra.run.smithi078.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-11T22:47:29.708 INFO:teuthology.orchestra.run.smithi078.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-11T22:47:29.708 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror.2] 2024-03-11T22:47:29.708 INFO:teuthology.orchestra.run.smithi078.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-11T22:47:29.708 INFO:teuthology.orchestra.run.smithi078.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-11T22:47:29.708 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror.3] 2024-03-11T22:47:29.708 INFO:teuthology.orchestra.run.smithi078.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-11T22:47:29.708 INFO:teuthology.orchestra.run.smithi078.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-11T22:47:29.716 INFO:teuthology.orchestra.run.smithi132.stdout:[global] 2024-03-11T22:47:29.716 INFO:teuthology.orchestra.run.smithi132.stdout: chdir = "" 2024-03-11T22:47:29.716 INFO:teuthology.orchestra.run.smithi132.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-11T22:47:29.716 INFO:teuthology.orchestra.run.smithi132.stdout: auth supported = cephx 2024-03-11T22:47:29.717 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:47:29.717 INFO:teuthology.orchestra.run.smithi132.stdout: filestore xattr use omap = true 2024-03-11T22:47:29.717 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:47:29.717 INFO:teuthology.orchestra.run.smithi132.stdout: mon clock drift allowed = 1.000 2024-03-11T22:47:29.717 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:47:29.717 INFO:teuthology.orchestra.run.smithi132.stdout: osd crush chooseleaf type = 0 2024-03-11T22:47:29.717 INFO:teuthology.orchestra.run.smithi132.stdout: auth debug = true 2024-03-11T22:47:29.717 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:47:29.717 INFO:teuthology.orchestra.run.smithi132.stdout: ms die on old message = true 2024-03-11T22:47:29.717 INFO:teuthology.orchestra.run.smithi132.stdout: ms die on bug = true 2024-03-11T22:47:29.717 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:47:29.717 INFO:teuthology.orchestra.run.smithi132.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-11T22:47:29.717 INFO:teuthology.orchestra.run.smithi132.stdout: mon pg warn max object skew = 0 2024-03-11T22:47:29.717 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:47:29.717 INFO:teuthology.orchestra.run.smithi132.stdout: # disable pg_autoscaler by default for new pools 2024-03-11T22:47:29.718 INFO:teuthology.orchestra.run.smithi132.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-11T22:47:29.718 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:47:29.718 INFO:teuthology.orchestra.run.smithi132.stdout: osd pool default size = 2 2024-03-11T22:47:29.718 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:47:29.718 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd allow primary affinity = true 2024-03-11T22:47:29.718 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd allow pg remap = true 2024-03-11T22:47:29.718 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on legacy crush tunables = false 2024-03-11T22:47:29.718 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on crush straw calc version zero = false 2024-03-11T22:47:29.718 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on no sortbitwise = false 2024-03-11T22:47:29.718 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on osd down out interval zero = false 2024-03-11T22:47:29.718 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on too few osds = false 2024-03-11T22:47:29.718 INFO:teuthology.orchestra.run.smithi132.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-11T22:47:29.718 INFO:teuthology.orchestra.run.smithi132.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-11T22:47:29.718 INFO:teuthology.orchestra.run.smithi132.stdout: mon_allow_pool_size_one = true 2024-03-11T22:47:29.719 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:47:29.719 INFO:teuthology.orchestra.run.smithi132.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-11T22:47:29.719 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:47:29.719 INFO:teuthology.orchestra.run.smithi132.stdout: osd default data pool replay window = 5 2024-03-11T22:47:29.719 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:47:29.719 INFO:teuthology.orchestra.run.smithi132.stdout: mon allow pool delete = true 2024-03-11T22:47:29.719 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:47:29.719 INFO:teuthology.orchestra.run.smithi132.stdout: mon cluster log file level = debug 2024-03-11T22:47:29.719 INFO:teuthology.orchestra.run.smithi132.stdout: debug asserts on shutdown = true 2024-03-11T22:47:29.719 INFO:teuthology.orchestra.run.smithi132.stdout: mon health detail to clog = false 2024-03-11T22:47:29.719 INFO:teuthology.orchestra.run.smithi132.stdout: mon host = 172.21.15.132 2024-03-11T22:47:29.719 INFO:teuthology.orchestra.run.smithi132.stdout: mon client directed command retry = 5 2024-03-11T22:47:29.719 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on pool no app = False 2024-03-11T22:47:29.719 INFO:teuthology.orchestra.run.smithi132.stdout: ms inject socket failures = 5000 2024-03-11T22:47:29.719 INFO:teuthology.orchestra.run.smithi132.stdout: fsid = f1daf4d2-a7c6-44cf-bbbf-27e068deb999 2024-03-11T22:47:29.720 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:47:29.720 INFO:teuthology.orchestra.run.smithi132.stdout:[osd] 2024-03-11T22:47:29.720 INFO:teuthology.orchestra.run.smithi132.stdout: osd journal size = 100 2024-03-11T22:47:29.720 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:47:29.720 INFO:teuthology.orchestra.run.smithi132.stdout: osd scrub load threshold = 5.0 2024-03-11T22:47:29.720 INFO:teuthology.orchestra.run.smithi132.stdout: osd scrub max interval = 600 2024-03-11T22:47:29.720 INFO:teuthology.orchestra.run.smithi132.stdout: osd mclock profile = high_recovery_ops 2024-03-11T22:47:29.720 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:47:29.720 INFO:teuthology.orchestra.run.smithi132.stdout: osd recover clone overlap = true 2024-03-11T22:47:29.720 INFO:teuthology.orchestra.run.smithi132.stdout: osd recovery max chunk = 1048576 2024-03-11T22:47:29.720 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:47:29.720 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug shutdown = true 2024-03-11T22:47:29.720 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug op order = true 2024-03-11T22:47:29.720 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug verify stray on activate = true 2024-03-11T22:47:29.720 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug trim objects = true 2024-03-11T22:47:29.721 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:47:29.721 INFO:teuthology.orchestra.run.smithi132.stdout: osd open classes on start = true 2024-03-11T22:47:29.721 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug pg log writeout = true 2024-03-11T22:47:29.721 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:47:29.721 INFO:teuthology.orchestra.run.smithi132.stdout: osd deep scrub update digest min age = 30 2024-03-11T22:47:29.721 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:47:29.721 INFO:teuthology.orchestra.run.smithi132.stdout: osd map max advance = 10 2024-03-11T22:47:29.721 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:47:29.721 INFO:teuthology.orchestra.run.smithi132.stdout: journal zero on create = true 2024-03-11T22:47:29.721 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:47:29.721 INFO:teuthology.orchestra.run.smithi132.stdout: filestore ondisk finisher threads = 3 2024-03-11T22:47:29.721 INFO:teuthology.orchestra.run.smithi132.stdout: filestore apply finisher threads = 3 2024-03-11T22:47:29.721 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:47:29.721 INFO:teuthology.orchestra.run.smithi132.stdout: bdev debug aio = true 2024-03-11T22:47:29.722 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug misdirected ops = true 2024-03-11T22:47:29.722 INFO:teuthology.orchestra.run.smithi132.stdout: bdev async discard = True 2024-03-11T22:47:29.722 INFO:teuthology.orchestra.run.smithi132.stdout: bdev enable discard = True 2024-03-11T22:47:29.722 INFO:teuthology.orchestra.run.smithi132.stdout: bluestore allocator = stupid 2024-03-11T22:47:29.722 INFO:teuthology.orchestra.run.smithi132.stdout: bluestore block size = 96636764160 2024-03-11T22:47:29.722 INFO:teuthology.orchestra.run.smithi132.stdout: bluestore fsck on mount = True 2024-03-11T22:47:29.722 INFO:teuthology.orchestra.run.smithi132.stdout: debug bluefs = 1/20 2024-03-11T22:47:29.722 INFO:teuthology.orchestra.run.smithi132.stdout: debug bluestore = 1/20 2024-03-11T22:47:29.722 INFO:teuthology.orchestra.run.smithi132.stdout: debug ms = 1 2024-03-11T22:47:29.722 INFO:teuthology.orchestra.run.smithi132.stdout: debug osd = 20 2024-03-11T22:47:29.722 INFO:teuthology.orchestra.run.smithi132.stdout: debug rocksdb = 4/10 2024-03-11T22:47:29.722 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd backfillfull_ratio = 0.85 2024-03-11T22:47:29.722 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd full ratio = 0.9 2024-03-11T22:47:29.722 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd nearfull ratio = 0.8 2024-03-11T22:47:29.723 INFO:teuthology.orchestra.run.smithi132.stdout: osd failsafe full ratio = 0.95 2024-03-11T22:47:29.723 INFO:teuthology.orchestra.run.smithi132.stdout: osd objectstore = bluestore 2024-03-11T22:47:29.723 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:47:29.723 INFO:teuthology.orchestra.run.smithi132.stdout:[mgr] 2024-03-11T22:47:29.723 INFO:teuthology.orchestra.run.smithi132.stdout: debug ms = 1 2024-03-11T22:47:29.723 INFO:teuthology.orchestra.run.smithi132.stdout: debug mgr = 20 2024-03-11T22:47:29.723 INFO:teuthology.orchestra.run.smithi132.stdout: debug mon = 20 2024-03-11T22:47:29.723 INFO:teuthology.orchestra.run.smithi132.stdout: debug auth = 20 2024-03-11T22:47:29.723 INFO:teuthology.orchestra.run.smithi132.stdout: mon reweight min pgs per osd = 4 2024-03-11T22:47:29.723 INFO:teuthology.orchestra.run.smithi132.stdout: mon reweight min bytes per osd = 10 2024-03-11T22:47:29.723 INFO:teuthology.orchestra.run.smithi132.stdout: mgr/telemetry/nag = false 2024-03-11T22:47:29.723 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:47:29.723 INFO:teuthology.orchestra.run.smithi132.stdout:[mon] 2024-03-11T22:47:29.723 INFO:teuthology.orchestra.run.smithi132.stdout: debug ms = 1 2024-03-11T22:47:29.723 INFO:teuthology.orchestra.run.smithi132.stdout: debug mon = 20 2024-03-11T22:47:29.723 INFO:teuthology.orchestra.run.smithi132.stdout: debug paxos = 20 2024-03-11T22:47:29.724 INFO:teuthology.orchestra.run.smithi132.stdout: debug auth = 20 2024-03-11T22:47:29.724 INFO:teuthology.orchestra.run.smithi132.stdout: mon data avail warn = 5 2024-03-11T22:47:29.724 INFO:teuthology.orchestra.run.smithi132.stdout: mon mgr mkfs grace = 240 2024-03-11T22:47:29.724 INFO:teuthology.orchestra.run.smithi132.stdout: mon reweight min pgs per osd = 4 2024-03-11T22:47:29.724 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd reporter subtree level = osd 2024-03-11T22:47:29.724 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd prime pg temp = true 2024-03-11T22:47:29.724 INFO:teuthology.orchestra.run.smithi132.stdout: mon reweight min bytes per osd = 10 2024-03-11T22:47:29.724 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:47:29.724 INFO:teuthology.orchestra.run.smithi132.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-11T22:47:29.724 INFO:teuthology.orchestra.run.smithi132.stdout: auth mon ticket ttl = 660 # 11m 2024-03-11T22:47:29.724 INFO:teuthology.orchestra.run.smithi132.stdout: auth service ticket ttl = 240 # 4m 2024-03-11T22:47:29.724 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:47:29.724 INFO:teuthology.orchestra.run.smithi132.stdout: # don't complain about insecure global_id in the test suite 2024-03-11T22:47:29.724 INFO:teuthology.orchestra.run.smithi132.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-11T22:47:29.724 INFO:teuthology.orchestra.run.smithi132.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-11T22:47:29.725 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:47:29.725 INFO:teuthology.orchestra.run.smithi132.stdout: # 1m isn't quite enough 2024-03-11T22:47:29.725 INFO:teuthology.orchestra.run.smithi132.stdout: mon_down_mkfs_grace = 2m 2024-03-11T22:47:29.725 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:47:29.725 INFO:teuthology.orchestra.run.smithi132.stdout: mon_warn_on_filestore_osds = false 2024-03-11T22:47:29.725 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:47:29.726 INFO:teuthology.orchestra.run.smithi132.stdout:[client] 2024-03-11T22:47:29.726 INFO:teuthology.orchestra.run.smithi132.stdout: rgw cache enabled = true 2024-03-11T22:47:29.726 INFO:teuthology.orchestra.run.smithi132.stdout: rgw enable ops log = true 2024-03-11T22:47:29.726 INFO:teuthology.orchestra.run.smithi132.stdout: rgw enable usage log = true 2024-03-11T22:47:29.726 INFO:teuthology.orchestra.run.smithi132.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-11T22:47:29.726 INFO:teuthology.orchestra.run.smithi132.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-11T22:47:29.727 INFO:teuthology.orchestra.run.smithi132.stdout: debug rbd = 20 2024-03-11T22:47:29.727 INFO:teuthology.orchestra.run.smithi132.stdout: debug rbd_mirror = 15 2024-03-11T22:47:29.727 INFO:teuthology.orchestra.run.smithi132.stdout: log to stderr = False 2024-03-11T22:47:29.727 INFO:teuthology.orchestra.run.smithi132.stdout: rbd default features = 125 2024-03-11T22:47:29.727 INFO:teuthology.orchestra.run.smithi132.stdout: rbd mirror image policy type = simple 2024-03-11T22:47:29.727 INFO:teuthology.orchestra.run.smithi132.stdout:[mon.a] 2024-03-11T22:47:29.728 INFO:teuthology.orchestra.run.smithi132.stdout:[client.mirror.0] 2024-03-11T22:47:29.728 INFO:teuthology.orchestra.run.smithi132.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-11T22:47:29.728 INFO:teuthology.orchestra.run.smithi132.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-11T22:47:29.728 INFO:teuthology.orchestra.run.smithi132.stdout:[client.mirror.1] 2024-03-11T22:47:29.728 INFO:teuthology.orchestra.run.smithi132.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-11T22:47:29.728 INFO:teuthology.orchestra.run.smithi132.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-11T22:47:29.728 INFO:teuthology.orchestra.run.smithi132.stdout:[client.mirror.2] 2024-03-11T22:47:29.728 INFO:teuthology.orchestra.run.smithi132.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-11T22:47:29.728 INFO:teuthology.orchestra.run.smithi132.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-11T22:47:29.728 INFO:teuthology.orchestra.run.smithi132.stdout:[client.mirror.3] 2024-03-11T22:47:29.728 INFO:teuthology.orchestra.run.smithi132.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-11T22:47:29.728 INFO:teuthology.orchestra.run.smithi132.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-11T22:47:29.731 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2024-03-11T22:47:29.731 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2024-03-11T22:47:29.826 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-11T22:47:29.827 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:47:29.827 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2024-03-11T22:47:29.841 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:47:29.842 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2024-03-11T22:47:29.896 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi078.front.sepia.ceph.com 2024-03-11T22:47:29.896 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-11T22:47:29.897 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/cluster2.keyring 2024-03-11T22:47:29.897 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-03-11T22:47:29.937 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-11T22:47:29.937 DEBUG:teuthology.orchestra.run.smithi078:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-03-11T22:47:29.991 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi132.front.sepia.ceph.com 2024-03-11T22:47:29.992 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:47:29.992 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/ceph/cluster2.keyring 2024-03-11T22:47:29.992 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-03-11T22:47:30.032 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:47:30.032 DEBUG:teuthology.orchestra.run.smithi132:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-03-11T22:47:30.087 INFO:tasks.ceph:Setting up mon nodes... 2024-03-11T22:47:30.087 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-11T22:47:30.087 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2024-03-11T22:47:30.153 INFO:teuthology.orchestra.run.smithi078.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2024-03-11T22:47:30.155 INFO:tasks.ceph:Setting up mds nodes... 2024-03-11T22:47:30.155 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-11T22:47:30.156 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2024-03-11T22:47:30.197 INFO:teuthology.orchestra.run.smithi078.stdout:creating /etc/ceph/cluster2.client.0.keyring 2024-03-11T22:47:30.213 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2024-03-11T22:47:30.255 INFO:teuthology.orchestra.run.smithi132.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2024-03-11T22:47:30.271 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2024-03-11T22:47:30.352 INFO:teuthology.orchestra.run.smithi132.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2024-03-11T22:47:30.368 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2024-03-11T22:47:30.450 INFO:teuthology.orchestra.run.smithi132.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2024-03-11T22:47:30.465 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2024-03-11T22:47:30.548 INFO:teuthology.orchestra.run.smithi132.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2024-03-11T22:47:30.563 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2024-03-11T22:47:30.645 INFO:teuthology.orchestra.run.smithi132.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2024-03-11T22:47:30.660 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2024-03-11T22:47:30.742 INFO:teuthology.orchestra.run.smithi132.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2024-03-11T22:47:30.757 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2024-03-11T22:47:30.838 INFO:teuthology.orchestra.run.smithi132.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2024-03-11T22:47:30.854 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2024-03-11T22:47:30.936 INFO:teuthology.orchestra.run.smithi132.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2024-03-11T22:47:30.951 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-11T22:47:30.952 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi078.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi132.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-03-11T22:47:30.952 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2024-03-11T22:47:31.018 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-03-11T22:47:31.018 INFO:tasks.ceph:role: cluster2.osd.0 2024-03-11T22:47:31.019 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi132.front.sepia.ceph.com 2024-03-11T22:47:31.019 DEBUG:teuthology.orchestra.run.smithi132:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-11T22:47:31.093 INFO:teuthology.orchestra.run.smithi132.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-11T22:47:31.093 INFO:teuthology.orchestra.run.smithi132.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-11T22:47:31.093 INFO:teuthology.orchestra.run.smithi132.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-11T22:47:31.093 INFO:teuthology.orchestra.run.smithi132.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-11T22:47:31.093 INFO:teuthology.orchestra.run.smithi132.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-11T22:47:31.094 INFO:teuthology.orchestra.run.smithi132.stdout: = sunit=0 swidth=0 blks 2024-03-11T22:47:31.094 INFO:teuthology.orchestra.run.smithi132.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-11T22:47:31.094 INFO:teuthology.orchestra.run.smithi132.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-11T22:47:31.094 INFO:teuthology.orchestra.run.smithi132.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-11T22:47:31.094 INFO:teuthology.orchestra.run.smithi132.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-11T22:47:31.606 INFO:teuthology.orchestra.run.smithi132.stdout:Discarding blocks...Done. 2024-03-11T22:47:31.607 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi132.front.sepia.ceph.com -o noatime 2024-03-11T22:47:31.608 DEBUG:teuthology.orchestra.run.smithi132:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2024-03-11T22:47:31.752 DEBUG:teuthology.orchestra.run.smithi132:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2024-03-11T22:47:31.786 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2024-03-11T22:47:31.855 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-03-11T22:47:31.855 INFO:tasks.ceph:role: cluster2.osd.1 2024-03-11T22:47:31.855 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi132.front.sepia.ceph.com 2024-03-11T22:47:31.855 DEBUG:teuthology.orchestra.run.smithi132:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-11T22:47:31.929 INFO:teuthology.orchestra.run.smithi132.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-11T22:47:31.930 INFO:teuthology.orchestra.run.smithi132.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-11T22:47:31.930 INFO:teuthology.orchestra.run.smithi132.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-11T22:47:31.930 INFO:teuthology.orchestra.run.smithi132.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-11T22:47:31.930 INFO:teuthology.orchestra.run.smithi132.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-11T22:47:31.930 INFO:teuthology.orchestra.run.smithi132.stdout: = sunit=0 swidth=0 blks 2024-03-11T22:47:31.930 INFO:teuthology.orchestra.run.smithi132.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-11T22:47:31.930 INFO:teuthology.orchestra.run.smithi132.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-11T22:47:31.930 INFO:teuthology.orchestra.run.smithi132.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-11T22:47:31.930 INFO:teuthology.orchestra.run.smithi132.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-11T22:47:32.376 INFO:teuthology.orchestra.run.smithi132.stdout:Discarding blocks...Done. 2024-03-11T22:47:32.378 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi132.front.sepia.ceph.com -o noatime 2024-03-11T22:47:32.378 DEBUG:teuthology.orchestra.run.smithi132:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2024-03-11T22:47:32.465 DEBUG:teuthology.orchestra.run.smithi132:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2024-03-11T22:47:32.496 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2024-03-11T22:47:32.566 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-03-11T22:47:32.566 INFO:tasks.ceph:role: cluster2.osd.2 2024-03-11T22:47:32.567 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi132.front.sepia.ceph.com 2024-03-11T22:47:32.567 DEBUG:teuthology.orchestra.run.smithi132:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-11T22:47:32.641 INFO:teuthology.orchestra.run.smithi132.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-11T22:47:32.642 INFO:teuthology.orchestra.run.smithi132.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-11T22:47:32.642 INFO:teuthology.orchestra.run.smithi132.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-11T22:47:32.642 INFO:teuthology.orchestra.run.smithi132.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-11T22:47:32.642 INFO:teuthology.orchestra.run.smithi132.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-11T22:47:32.642 INFO:teuthology.orchestra.run.smithi132.stdout: = sunit=0 swidth=0 blks 2024-03-11T22:47:32.642 INFO:teuthology.orchestra.run.smithi132.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-11T22:47:32.643 INFO:teuthology.orchestra.run.smithi132.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-11T22:47:32.643 INFO:teuthology.orchestra.run.smithi132.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-11T22:47:32.643 INFO:teuthology.orchestra.run.smithi132.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-11T22:47:33.133 INFO:teuthology.orchestra.run.smithi132.stdout:Discarding blocks...Done. 2024-03-11T22:47:33.135 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi132.front.sepia.ceph.com -o noatime 2024-03-11T22:47:33.136 DEBUG:teuthology.orchestra.run.smithi132:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2024-03-11T22:47:33.277 DEBUG:teuthology.orchestra.run.smithi132:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2024-03-11T22:47:33.312 DEBUG:teuthology.orchestra.run.smithi132:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-03-11T22:47:33.410 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-11T22:47:33.403+0000 7fc588037640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2024-03-11T22:47:33.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-11T22:47:33.407+0000 7fc588037640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2024-03-11T22:47:33.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-11T22:47:33.407+0000 7fc588037640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-03-11T22:47:33.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-11T22:47:33.407+0000 7fc588037640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-03-11T22:47:33.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-11T22:47:33.407+0000 7fc588037640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-03-11T22:47:33.423 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-11T22:47:33.416+0000 7fc588037640 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2024-03-11T22:47:36.279 DEBUG:teuthology.orchestra.run.smithi132:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2024-03-11T22:47:36.311 DEBUG:teuthology.orchestra.run.smithi132:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-03-11T22:47:36.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-11T22:47:36.398+0000 7fec47338640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2024-03-11T22:47:36.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-11T22:47:36.398+0000 7fec47338640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2024-03-11T22:47:36.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-11T22:47:36.398+0000 7fec47338640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-03-11T22:47:36.406 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-11T22:47:36.398+0000 7fec47338640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-03-11T22:47:36.406 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-11T22:47:36.398+0000 7fec47338640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-03-11T22:47:36.406 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-11T22:47:36.399+0000 7fec47338640 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2024-03-11T22:47:39.259 DEBUG:teuthology.orchestra.run.smithi132:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2024-03-11T22:47:39.291 DEBUG:teuthology.orchestra.run.smithi132:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-03-11T22:47:39.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-11T22:47:39.374+0000 7f0627222640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2024-03-11T22:47:39.382 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-11T22:47:39.374+0000 7f0627222640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2024-03-11T22:47:39.382 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-11T22:47:39.374+0000 7f0627222640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-03-11T22:47:39.382 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-11T22:47:39.374+0000 7f0627222640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-03-11T22:47:39.382 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-11T22:47:39.374+0000 7f0627222640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-03-11T22:47:39.382 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-11T22:47:39.375+0000 7f0627222640 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2024-03-11T22:47:42.238 DEBUG:teuthology.orchestra.run.smithi132:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2024-03-11T22:47:42.269 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-11T22:47:42.270 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-11T22:47:42.270 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2024-03-11T22:47:42.302 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:47:42.302 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2024-03-11T22:47:42.339 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:47:42.340 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2024-03-11T22:47:42.409 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:47:42.409 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2024-03-11T22:47:42.477 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-11T22:47:42.477 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2024-03-11T22:47:42.495 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:47:42.495 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2024-03-11T22:47:42.534 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:47:42.534 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2024-03-11T22:47:42.589 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:47:42.589 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2024-03-11T22:47:42.645 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:47:42.645 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2024-03-11T22:47:42.701 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:47:42.701 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2024-03-11T22:47:42.757 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:47:42.757 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2024-03-11T22:47:42.814 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:47:42.814 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2024-03-11T22:47:42.869 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:47:42.869 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2024-03-11T22:47:42.925 INFO:tasks.ceph:Adding keys to all mons... 2024-03-11T22:47:42.926 DEBUG:teuthology.orchestra.run.smithi132:> sudo tee -a /etc/ceph/cluster2.keyring 2024-03-11T22:47:42.991 INFO:teuthology.orchestra.run.smithi132.stdout:[mgr.x] 2024-03-11T22:47:42.992 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQACiu9l/TDFCBAAZGfn6z90zs5mhg9290jFOA== 2024-03-11T22:47:42.992 INFO:teuthology.orchestra.run.smithi132.stdout:[osd.0] 2024-03-11T22:47:42.992 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQAFiu9lWSsYGBAAgT2+ZnYeMJPI49/zMlLMog== 2024-03-11T22:47:42.992 INFO:teuthology.orchestra.run.smithi132.stdout:[osd.1] 2024-03-11T22:47:42.992 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQAIiu9lcP3KFxAA7p64Mu38hC9cnE/e4IOc2Q== 2024-03-11T22:47:42.992 INFO:teuthology.orchestra.run.smithi132.stdout:[osd.2] 2024-03-11T22:47:42.993 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQALiu9l2kRfFhAA2/BouT5waoWW6RKawEzjUg== 2024-03-11T22:47:42.993 INFO:teuthology.orchestra.run.smithi132.stdout:[client.0] 2024-03-11T22:47:42.993 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQACiu9lgtdxCxAAlabqqotFeJfe7cjoEV8b8g== 2024-03-11T22:47:42.993 INFO:teuthology.orchestra.run.smithi132.stdout:[client.mirror] 2024-03-11T22:47:42.993 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQACiu9lNtLYDhAAszjbT/to6zICOSn/nO3uFA== 2024-03-11T22:47:42.993 INFO:teuthology.orchestra.run.smithi132.stdout:[client.mirror.0] 2024-03-11T22:47:42.993 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQACiu9l9VyrFBAAQkuCTDjv/iLToWza0ePB5w== 2024-03-11T22:47:42.994 INFO:teuthology.orchestra.run.smithi132.stdout:[client.mirror.1] 2024-03-11T22:47:42.994 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQACiu9l9W1/GhAAxNc3mhPRr7Q/LtK4nzxbdQ== 2024-03-11T22:47:42.994 INFO:teuthology.orchestra.run.smithi132.stdout:[client.mirror.2] 2024-03-11T22:47:42.994 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQACiu9lwmRVIBAAKfW08bQoLBKly5+7lfBrrQ== 2024-03-11T22:47:42.994 INFO:teuthology.orchestra.run.smithi132.stdout:[client.mirror.3] 2024-03-11T22:47:42.994 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQACiu9lAsAbJhAAoVbqNNiPS8wYxf6YXjQ+Pg== 2024-03-11T22:47:42.994 INFO:teuthology.orchestra.run.smithi132.stdout:[client.mirror.4] 2024-03-11T22:47:42.995 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQACiu9l5cDnKxAAQ1a6D1PjpRJ2MK1DnA/0Iw== 2024-03-11T22:47:42.995 INFO:teuthology.orchestra.run.smithi132.stdout:[client.mirror.5] 2024-03-11T22:47:42.995 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQACiu9lyCGlMRAALPb5oclxBp4bk2kCE4hLsw== 2024-03-11T22:47:42.995 INFO:teuthology.orchestra.run.smithi132.stdout:[client.mirror.6] 2024-03-11T22:47:42.995 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQACiu9lLpN0NxAACSsEU9izu/9++E3MmxrDAQ== 2024-03-11T22:47:42.996 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-11T22:47:43.090 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-11T22:47:43.142 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-11T22:47:43.194 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-11T22:47:43.245 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-11T22:47:43.299 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-11T22:47:43.353 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-11T22:47:43.407 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-11T22:47:43.462 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-11T22:47:43.517 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-11T22:47:43.571 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-11T22:47:43.627 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-11T22:47:43.682 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-11T22:47:43.736 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-11T22:47:43.736 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2024-03-11T22:47:43.763 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2024-03-11T22:47:43.860 DEBUG:teuthology.orchestra.run.smithi132:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2024-03-11T22:47:43.887 DEBUG:teuthology.orchestra.run.smithi132:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2024-03-11T22:47:43.941 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2024-03-11T22:47:43.942 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2024-03-11T22:47:43.942 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2024-03-11T22:47:43.984 INFO:tasks.ceph.cluster2.mon.a:Started 2024-03-11T22:47:43.984 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2024-03-11T22:47:43.985 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2024-03-11T22:47:43.985 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2024-03-11T22:47:43.987 INFO:tasks.ceph.cluster2.mgr.x:Started 2024-03-11T22:47:43.988 INFO:tasks.ceph:Setting crush tunables to default 2024-03-11T22:47:43.988 DEBUG:teuthology.orchestra.run.smithi132:> sudo ceph --cluster cluster2 osd crush tunables default 2024-03-11T22:47:44.176 INFO:teuthology.orchestra.run.smithi132.stderr:adjusted tunables profile to default 2024-03-11T22:47:44.187 INFO:tasks.ceph:check_enable_crimson: False 2024-03-11T22:47:44.187 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2024-03-11T22:47:44.187 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:47:44.187 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2024-03-11T22:47:44.213 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:47:44.214 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2024-03-11T22:47:44.282 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:47:44.282 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2024-03-11T22:47:44.348 DEBUG:teuthology.orchestra.run.smithi132:> sudo ceph --cluster cluster2 osd new fb0f38b9-fa30-4dd3-8c65-8df211337f59 0 2024-03-11T22:47:44.454 INFO:tasks.ceph.cluster2.mgr.x.smithi078.stderr:2024-03-11T22:47:44.447+0000 7fa6e9639040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T22:47:44.501 INFO:tasks.ceph.cluster2.mgr.x.smithi078.stderr:2024-03-11T22:47:44.494+0000 7fa6e9639040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T22:47:44.566 INFO:teuthology.orchestra.run.smithi132.stdout:0 2024-03-11T22:47:44.577 DEBUG:teuthology.orchestra.run.smithi132:> sudo ceph --cluster cluster2 osd new 35e93d66-5535-4e84-8f82-4722c0fe243e 1 2024-03-11T22:47:44.754 INFO:teuthology.orchestra.run.smithi132.stdout:1 2024-03-11T22:47:44.765 DEBUG:teuthology.orchestra.run.smithi132:> sudo ceph --cluster cluster2 osd new f5298c4e-da38-4f0a-a751-3cb7d1d15fd2 2 2024-03-11T22:47:44.884 INFO:tasks.ceph.cluster2.mgr.x.smithi078.stderr:2024-03-11T22:47:44.877+0000 7fa6e9639040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T22:47:44.928 INFO:tasks.ceph.cluster2.mgr.x.smithi078.stderr:2024-03-11T22:47:44.921+0000 7fa6e9639040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T22:47:44.941 INFO:teuthology.orchestra.run.smithi132.stdout:2 2024-03-11T22:47:44.952 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2024-03-11T22:47:44.953 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2024-03-11T22:47:44.955 INFO:tasks.ceph.cluster2.osd.0:Started 2024-03-11T22:47:44.955 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2024-03-11T22:47:44.955 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2024-03-11T22:47:44.957 INFO:tasks.ceph.cluster2.osd.1:Started 2024-03-11T22:47:44.957 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2024-03-11T22:47:44.957 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2024-03-11T22:47:44.959 INFO:tasks.ceph.cluster2.osd.2:Started 2024-03-11T22:47:44.959 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-03-11T22:47:44.970 INFO:tasks.ceph.cluster2.mgr.x.smithi078.stderr:2024-03-11T22:47:44.963+0000 7fa6e9639040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T22:47:45.021 INFO:tasks.ceph.cluster2.mgr.x.smithi078.stderr:2024-03-11T22:47:45.014+0000 7fa6e9639040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T22:47:45.062 INFO:tasks.ceph.cluster2.mgr.x.smithi078.stderr:2024-03-11T22:47:45.055+0000 7fa6e9639040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T22:47:45.132 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:47:45.132 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":5,"fsid":"f1daf4d2-a7c6-44cf-bbbf-27e068deb999","created":"2024-03-11T22:47:44.090611+0000","modified":"2024-03-11T22:47:44.935041+0000","last_up_change":"0.000000","last_in_change":"2024-03-11T22:47:44.935041+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fb0f38b9-fa30-4dd3-8c65-8df211337f59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35e93d66-5535-4e84-8f82-4722c0fe243e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f5298c4e-da38-4f0a-a751-3cb7d1d15fd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T22:47:45.143 INFO:tasks.ceph.ceph_manager.cluster2:[] 2024-03-11T22:47:45.143 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-11T22:47:45.174 INFO:tasks.ceph.cluster2.mgr.x.smithi078.stderr:2024-03-11T22:47:45.167+0000 7fa6e9639040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T22:47:45.226 INFO:tasks.ceph.cluster2.mgr.x.smithi078.stderr:2024-03-11T22:47:45.219+0000 7fa6e9639040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T22:47:45.268 INFO:tasks.ceph.cluster2.mgr.x.smithi078.stderr:2024-03-11T22:47:45.261+0000 7fa6e9639040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T22:47:45.316 INFO:tasks.ceph.cluster2.mgr.x.smithi078.stderr:2024-03-11T22:47:45.309+0000 7fa6e9639040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T22:47:45.358 INFO:tasks.ceph.cluster2.mgr.x.smithi078.stderr:2024-03-11T22:47:45.351+0000 7fa6e9639040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T22:47:45.446 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-03-11T22:47:45.553 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr:2024-03-11T22:47:45.546+0000 7fc2710a6640 -1 Falling back to public interface 2024-03-11T22:47:45.554 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr:2024-03-11T22:47:45.546+0000 7f4dd38b3640 -1 Falling back to public interface 2024-03-11T22:47:45.554 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr:2024-03-11T22:47:45.546+0000 7f4dd2862640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6800/0: (98) Address already in use 2024-03-11T22:47:45.555 INFO:tasks.ceph.cluster2.osd.2.smithi132.stderr:2024-03-11T22:47:45.548+0000 7f6387c10640 -1 Falling back to public interface 2024-03-11T22:47:45.605 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-11T22:47:45.605 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":5,"fsid":"f1daf4d2-a7c6-44cf-bbbf-27e068deb999","created":"2024-03-11T22:47:44.090611+0000","modified":"2024-03-11T22:47:44.935041+0000","last_up_change":"0.000000","last_in_change":"2024-03-11T22:47:44.935041+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fb0f38b9-fa30-4dd3-8c65-8df211337f59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35e93d66-5535-4e84-8f82-4722c0fe243e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f5298c4e-da38-4f0a-a751-3cb7d1d15fd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T22:47:45.615 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-03-11T22:47:45.677 INFO:tasks.ceph.cluster2.mgr.x.smithi078.stderr:2024-03-11T22:47:45.670+0000 7fa6e9639040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T22:47:45.808 INFO:tasks.ceph.cluster2.mgr.x.smithi078.stderr:2024-03-11T22:47:45.801+0000 7fa6e9639040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T22:47:45.901 INFO:tasks.ceph.cluster2.mgr.x.smithi078.stderr:2024-03-11T22:47:45.894+0000 7fa6e9639040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T22:47:45.944 INFO:tasks.ceph.cluster2.mgr.x.smithi078.stderr:2024-03-11T22:47:45.937+0000 7fa6e9639040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T22:47:46.054 INFO:tasks.ceph.cluster2.mgr.x.smithi078.stderr:2024-03-11T22:47:46.047+0000 7fa6e9639040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:47:46.145 INFO:tasks.ceph.cluster2.mgr.x.smithi078.stderr:2024-03-11T22:47:46.138+0000 7fa6e9639040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T22:47:46.298 INFO:tasks.ceph.cluster2.mgr.x.smithi078.stderr:2024-03-11T22:47:46.291+0000 7fa6e9639040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:47:46.340 INFO:tasks.ceph.cluster2.mgr.x.smithi078.stderr:2024-03-11T22:47:46.333+0000 7fa6e9639040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T22:47:46.458 INFO:tasks.ceph.cluster2.mgr.x.smithi078.stderr:2024-03-11T22:47:46.451+0000 7fa6e9639040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T22:47:47.060 INFO:tasks.ceph.cluster2.mgr.x.smithi078.stderr:2024-03-11T22:47:47.053+0000 7fa6e9639040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T22:47:47.322 INFO:tasks.ceph.cluster2.mgr.x.smithi078.stderr:2024-03-11T22:47:47.315+0000 7fa6e9639040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T22:47:47.370 INFO:tasks.ceph.cluster2.mgr.x.smithi078.stderr:2024-03-11T22:47:47.363+0000 7fa6e9639040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T22:47:47.485 INFO:tasks.ceph.cluster2.mgr.x.smithi078.stderr:2024-03-11T22:47:47.479+0000 7fa6e9639040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T22:47:48.506 INFO:tasks.ceph.cluster2.mgr.x.smithi078.stderr:2024-03-11T22:47:48.499+0000 7fa68a3cd640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-03-11T22:47:49.031 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr:2024-03-11T22:47:49.024+0000 7fc2710a6640 -1 osd.0 0 log_to_monitors true 2024-03-11T22:47:49.058 INFO:tasks.ceph.cluster2.osd.2.smithi132.stderr:2024-03-11T22:47:49.051+0000 7f6387c10640 -1 osd.2 0 log_to_monitors true 2024-03-11T22:47:49.064 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr:2024-03-11T22:47:49.057+0000 7f4dd38b3640 -1 osd.1 0 log_to_monitors true 2024-03-11T22:47:51.919 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-03-11T22:47:52.330 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr:2024-03-11T22:47:52.323+0000 7f4dcf026640 -1 osd.1 0 waiting for initial osdmap 2024-03-11T22:47:52.342 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr:2024-03-11T22:47:52.335+0000 7f4dc7658640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:47:52.344 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-11T22:47:52.345 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":7,"fsid":"f1daf4d2-a7c6-44cf-bbbf-27e068deb999","created":"2024-03-11T22:47:44.090611+0000","modified":"2024-03-11T22:47:50.498479+0000","last_up_change":"0.000000","last_in_change":"2024-03-11T22:47:44.935041+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fb0f38b9-fa30-4dd3-8c65-8df211337f59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35e93d66-5535-4e84-8f82-4722c0fe243e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f5298c4e-da38-4f0a-a751-3cb7d1d15fd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T22:47:52.348 INFO:tasks.ceph.cluster2.osd.2.smithi132.stderr:2024-03-11T22:47:52.341+0000 7f6382dc1640 -1 osd.2 0 waiting for initial osdmap 2024-03-11T22:47:52.359 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-03-11T22:47:52.365 INFO:tasks.ceph.cluster2.osd.2.smithi132.stderr:2024-03-11T22:47:52.358+0000 7f637b3f3640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:47:52.365 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr:2024-03-11T22:47:52.358+0000 7fc26c819640 -1 osd.0 0 waiting for initial osdmap 2024-03-11T22:47:52.387 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr:2024-03-11T22:47:52.380+0000 7fc264e4b640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:47:58.662 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-03-11T22:47:58.947 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-11T22:47:58.947 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":11,"fsid":"f1daf4d2-a7c6-44cf-bbbf-27e068deb999","created":"2024-03-11T22:47:44.090611+0000","modified":"2024-03-11T22:47:56.516311+0000","last_up_change":"2024-03-11T22:47:53.342834+0000","last_in_change":"2024-03-11T22:47:44.935041+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T22:47:53.540135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fb0f38b9-fa30-4dd3-8c65-8df211337f59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":3998923596},{"type":"v1","addr":"172.21.15.132:6801","nonce":3998923596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6803","nonce":3998923596},{"type":"v1","addr":"172.21.15.132:6805","nonce":3998923596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6811","nonce":3998923596},{"type":"v1","addr":"172.21.15.132:6813","nonce":3998923596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6807","nonce":3998923596},{"type":"v1","addr":"172.21.15.132:6809","nonce":3998923596}]},"public_addr":"172.21.15.132:6801/3998923596","cluster_addr":"172.21.15.132:6805/3998923596","heartbeat_back_addr":"172.21.15.132:6813/3998923596","heartbeat_front_addr":"172.21.15.132:6809/3998923596","state":["exists","up"]},{"osd":1,"uuid":"35e93d66-5535-4e84-8f82-4722c0fe243e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":870091833},{"type":"v1","addr":"172.21.15.132:6804","nonce":870091833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":870091833},{"type":"v1","addr":"172.21.15.132:6808","nonce":870091833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":870091833},{"type":"v1","addr":"172.21.15.132:6815","nonce":870091833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":870091833},{"type":"v1","addr":"172.21.15.132:6812","nonce":870091833}]},"public_addr":"172.21.15.132:6804/870091833","cluster_addr":"172.21.15.132:6808/870091833","heartbeat_back_addr":"172.21.15.132:6815/870091833","heartbeat_front_addr":"172.21.15.132:6812/870091833","state":["exists","up"]},{"osd":2,"uuid":"f5298c4e-da38-4f0a-a751-3cb7d1d15fd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1312519933},{"type":"v1","addr":"172.21.15.132:6817","nonce":1312519933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1312519933},{"type":"v1","addr":"172.21.15.132:6819","nonce":1312519933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1312519933},{"type":"v1","addr":"172.21.15.132:6823","nonce":1312519933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1312519933},{"type":"v1","addr":"172.21.15.132:6821","nonce":1312519933}]},"public_addr":"172.21.15.132:6817/1312519933","cluster_addr":"172.21.15.132:6819/1312519933","heartbeat_back_addr":"172.21.15.132:6823/1312519933","heartbeat_front_addr":"172.21.15.132:6821/1312519933","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:47:49.991363+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:47:50.081258+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:47:50.029364+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T22:47:58.958 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-03-11T22:47:58.958 INFO:tasks.ceph:Creating RBD pool 2024-03-11T22:47:58.958 DEBUG:teuthology.orchestra.run.smithi132:> sudo ceph --cluster cluster2 osd pool create rbd 8 2024-03-11T22:47:59.541 INFO:teuthology.orchestra.run.smithi132.stderr:pool 'rbd' created 2024-03-11T22:47:59.556 DEBUG:teuthology.orchestra.run.smithi132:> sudo ceph --cluster cluster2 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-03-11T22:48:00.544 INFO:teuthology.orchestra.run.smithi132.stderr:enabled application 'rbd' on pool 'rbd' 2024-03-11T22:48:00.562 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2024-03-11T22:48:00.563 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2024-03-11T22:48:00.563 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2024-03-11T22:48:00.563 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2024-03-11T22:48:00.564 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-11T22:48:00.898 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:48:00.909 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":5,"flags":0,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":4275994807},{"type":"v1","addr":"172.21.15.78:6827","nonce":4275994807}]},"active_addr":"172.21.15.78:6827/4275994807","active_change":"2024-03-11T22:47:47.493026+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":3461034714}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":263328036}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":1136666124}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":3552844956}]}]} 2024-03-11T22:48:00.911 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2024-03-11T22:48:00.911 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2024-03-11T22:48:00.912 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-03-11T22:48:01.194 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:48:01.194 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":13,"fsid":"f1daf4d2-a7c6-44cf-bbbf-27e068deb999","created":"2024-03-11T22:47:44.090611+0000","modified":"2024-03-11T22:48:00.535134+0000","last_up_change":"2024-03-11T22:47:53.342834+0000","last_in_change":"2024-03-11T22:47:44.935041+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T22:47:53.540135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-11T22:47:59.256195+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fb0f38b9-fa30-4dd3-8c65-8df211337f59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":3998923596},{"type":"v1","addr":"172.21.15.132:6801","nonce":3998923596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6803","nonce":3998923596},{"type":"v1","addr":"172.21.15.132:6805","nonce":3998923596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6811","nonce":3998923596},{"type":"v1","addr":"172.21.15.132:6813","nonce":3998923596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6807","nonce":3998923596},{"type":"v1","addr":"172.21.15.132:6809","nonce":3998923596}]},"public_addr":"172.21.15.132:6801/3998923596","cluster_addr":"172.21.15.132:6805/3998923596","heartbeat_back_addr":"172.21.15.132:6813/3998923596","heartbeat_front_addr":"172.21.15.132:6809/3998923596","state":["exists","up"]},{"osd":1,"uuid":"35e93d66-5535-4e84-8f82-4722c0fe243e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":870091833},{"type":"v1","addr":"172.21.15.132:6804","nonce":870091833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":870091833},{"type":"v1","addr":"172.21.15.132:6808","nonce":870091833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":870091833},{"type":"v1","addr":"172.21.15.132:6815","nonce":870091833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":870091833},{"type":"v1","addr":"172.21.15.132:6812","nonce":870091833}]},"public_addr":"172.21.15.132:6804/870091833","cluster_addr":"172.21.15.132:6808/870091833","heartbeat_back_addr":"172.21.15.132:6815/870091833","heartbeat_front_addr":"172.21.15.132:6812/870091833","state":["exists","up"]},{"osd":2,"uuid":"f5298c4e-da38-4f0a-a751-3cb7d1d15fd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1312519933},{"type":"v1","addr":"172.21.15.132:6817","nonce":1312519933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1312519933},{"type":"v1","addr":"172.21.15.132:6819","nonce":1312519933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1312519933},{"type":"v1","addr":"172.21.15.132:6823","nonce":1312519933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1312519933},{"type":"v1","addr":"172.21.15.132:6821","nonce":1312519933}]},"public_addr":"172.21.15.132:6817/1312519933","cluster_addr":"172.21.15.132:6819/1312519933","heartbeat_back_addr":"172.21.15.132:6823/1312519933","heartbeat_front_addr":"172.21.15.132:6821/1312519933","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:47:49.991363+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:47:50.081258+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:47:50.029364+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T22:48:01.206 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2024-03-11T22:48:01.207 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-03-11T22:48:01.489 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:48:01.490 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":13,"fsid":"f1daf4d2-a7c6-44cf-bbbf-27e068deb999","created":"2024-03-11T22:47:44.090611+0000","modified":"2024-03-11T22:48:00.535134+0000","last_up_change":"2024-03-11T22:47:53.342834+0000","last_in_change":"2024-03-11T22:47:44.935041+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T22:47:53.540135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-11T22:47:59.256195+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fb0f38b9-fa30-4dd3-8c65-8df211337f59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":3998923596},{"type":"v1","addr":"172.21.15.132:6801","nonce":3998923596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6803","nonce":3998923596},{"type":"v1","addr":"172.21.15.132:6805","nonce":3998923596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6811","nonce":3998923596},{"type":"v1","addr":"172.21.15.132:6813","nonce":3998923596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6807","nonce":3998923596},{"type":"v1","addr":"172.21.15.132:6809","nonce":3998923596}]},"public_addr":"172.21.15.132:6801/3998923596","cluster_addr":"172.21.15.132:6805/3998923596","heartbeat_back_addr":"172.21.15.132:6813/3998923596","heartbeat_front_addr":"172.21.15.132:6809/3998923596","state":["exists","up"]},{"osd":1,"uuid":"35e93d66-5535-4e84-8f82-4722c0fe243e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":870091833},{"type":"v1","addr":"172.21.15.132:6804","nonce":870091833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":870091833},{"type":"v1","addr":"172.21.15.132:6808","nonce":870091833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":870091833},{"type":"v1","addr":"172.21.15.132:6815","nonce":870091833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":870091833},{"type":"v1","addr":"172.21.15.132:6812","nonce":870091833}]},"public_addr":"172.21.15.132:6804/870091833","cluster_addr":"172.21.15.132:6808/870091833","heartbeat_back_addr":"172.21.15.132:6815/870091833","heartbeat_front_addr":"172.21.15.132:6812/870091833","state":["exists","up"]},{"osd":2,"uuid":"f5298c4e-da38-4f0a-a751-3cb7d1d15fd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1312519933},{"type":"v1","addr":"172.21.15.132:6817","nonce":1312519933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1312519933},{"type":"v1","addr":"172.21.15.132:6819","nonce":1312519933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1312519933},{"type":"v1","addr":"172.21.15.132:6823","nonce":1312519933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1312519933},{"type":"v1","addr":"172.21.15.132:6821","nonce":1312519933}]},"public_addr":"172.21.15.132:6817/1312519933","cluster_addr":"172.21.15.132:6819/1312519933","heartbeat_back_addr":"172.21.15.132:6823/1312519933","heartbeat_front_addr":"172.21.15.132:6821/1312519933","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:47:49.991363+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:47:50.081258+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:47:50.029364+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T22:48:01.502 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2024-03-11T22:48:01.503 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2024-03-11T22:48:01.503 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2024-03-11T22:48:01.645 INFO:teuthology.orchestra.run.smithi132.stdout:34359738374 2024-03-11T22:48:01.645 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-03-11T22:48:01.646 INFO:teuthology.orchestra.run.smithi132.stdout:34359738374 2024-03-11T22:48:01.646 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-03-11T22:48:01.647 INFO:teuthology.orchestra.run.smithi132.stdout:34359738374 2024-03-11T22:48:01.648 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-03-11T22:48:01.993 INFO:teuthology.orchestra.run.smithi132.stdout:34359738373 2024-03-11T22:48:02.005 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.0 2024-03-11T22:48:02.007 INFO:teuthology.orchestra.run.smithi132.stdout:34359738373 2024-03-11T22:48:02.007 INFO:teuthology.orchestra.run.smithi132.stdout:34359738373 2024-03-11T22:48:02.021 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.1 2024-03-11T22:48:02.022 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.2 2024-03-11T22:48:03.007 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-03-11T22:48:03.022 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-03-11T22:48:03.022 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-03-11T22:48:03.323 INFO:teuthology.orchestra.run.smithi132.stdout:34359738373 2024-03-11T22:48:03.340 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.0 2024-03-11T22:48:03.360 INFO:teuthology.orchestra.run.smithi132.stdout:34359738373 2024-03-11T22:48:03.360 INFO:teuthology.orchestra.run.smithi132.stdout:34359738373 2024-03-11T22:48:03.400 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.1 2024-03-11T22:48:03.400 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.2 2024-03-11T22:48:04.341 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-03-11T22:48:04.400 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-03-11T22:48:04.401 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-03-11T22:48:04.658 INFO:teuthology.orchestra.run.smithi132.stdout:34359738374 2024-03-11T22:48:04.670 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.0 2024-03-11T22:48:04.671 DEBUG:teuthology.parallel:result is None 2024-03-11T22:48:04.704 INFO:teuthology.orchestra.run.smithi132.stdout:34359738374 2024-03-11T22:48:04.707 INFO:teuthology.orchestra.run.smithi132.stdout:34359738374 2024-03-11T22:48:04.718 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.1 2024-03-11T22:48:04.719 DEBUG:teuthology.parallel:result is None 2024-03-11T22:48:04.720 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.2 2024-03-11T22:48:04.720 DEBUG:teuthology.parallel:result is None 2024-03-11T22:48:04.720 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-03-11T22:48:04.720 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-03-11T22:48:05.043 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:48:05.043 INFO:teuthology.orchestra.run.smithi132.stderr:dumped all 2024-03-11T22:48:05.055 INFO:teuthology.orchestra.run.smithi132.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-03-11T22:48:03.499335+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82556,"kb_used_data":1496,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032964,"statfs":{"total":289910292480,"available":289825755136,"internally_reserved":0,"allocated":1531904,"data_stored":1274801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000615"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-11T22:48:00.547691+0000","last_change":"2024-03-11T22:48:00.545536+0000","last_active":"2024-03-11T22:48:00.547691+0000","last_peered":"2024-03-11T22:48:00.547691+0000","last_clean":"2024-03-11T22:48:00.547691+0000","last_became_active":"2024-03-11T22:48:00.545246+0000","last_became_peered":"2024-03-11T22:48:00.545246+0000","last_unstale":"2024-03-11T22:48:00.547691+0000","last_undegraded":"2024-03-11T22:48:00.547691+0000","last_fullsized":"2024-03-11T22:48:00.547691+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:47:59.528950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:47:59.528950+0000","last_clean_scrub_stamp":"2024-03-11T22:47:59.528950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:14:24.117030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-11T22:48:00.548516+0000","last_change":"2024-03-11T22:48:00.547135+0000","last_active":"2024-03-11T22:48:00.548516+0000","last_peered":"2024-03-11T22:48:00.548516+0000","last_clean":"2024-03-11T22:48:00.548516+0000","last_became_active":"2024-03-11T22:48:00.546044+0000","last_became_peered":"2024-03-11T22:48:00.546044+0000","last_unstale":"2024-03-11T22:48:00.548516+0000","last_undegraded":"2024-03-11T22:48:00.548516+0000","last_fullsized":"2024-03-11T22:48:00.548516+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:47:59.528950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:47:59.528950+0000","last_clean_scrub_stamp":"2024-03-11T22:47:59.528950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:20:22.456780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-11T22:48:00.547762+0000","last_change":"2024-03-11T22:48:00.545568+0000","last_active":"2024-03-11T22:48:00.547762+0000","last_peered":"2024-03-11T22:48:00.547762+0000","last_clean":"2024-03-11T22:48:00.547762+0000","last_became_active":"2024-03-11T22:48:00.545283+0000","last_became_peered":"2024-03-11T22:48:00.545283+0000","last_unstale":"2024-03-11T22:48:00.547762+0000","last_undegraded":"2024-03-11T22:48:00.547762+0000","last_fullsized":"2024-03-11T22:48:00.547762+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:47:59.528950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:47:59.528950+0000","last_clean_scrub_stamp":"2024-03-11T22:47:59.528950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:05:32.693329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-11T22:48:00.548415+0000","last_change":"2024-03-11T22:48:00.546690+0000","last_active":"2024-03-11T22:48:00.548415+0000","last_peered":"2024-03-11T22:48:00.548415+0000","last_clean":"2024-03-11T22:48:00.548415+0000","last_became_active":"2024-03-11T22:48:00.546422+0000","last_became_peered":"2024-03-11T22:48:00.546422+0000","last_unstale":"2024-03-11T22:48:00.548415+0000","last_undegraded":"2024-03-11T22:48:00.548415+0000","last_fullsized":"2024-03-11T22:48:00.548415+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:47:59.528950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:47:59.528950+0000","last_clean_scrub_stamp":"2024-03-11T22:47:59.528950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:14:52.729483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-11T22:48:01.544086+0000","last_change":"2024-03-11T22:48:00.547841+0000","last_active":"2024-03-11T22:48:01.544086+0000","last_peered":"2024-03-11T22:48:01.544086+0000","last_clean":"2024-03-11T22:48:01.544086+0000","last_became_active":"2024-03-11T22:48:00.547567+0000","last_became_peered":"2024-03-11T22:48:00.547567+0000","last_unstale":"2024-03-11T22:48:01.544086+0000","last_undegraded":"2024-03-11T22:48:01.544086+0000","last_fullsized":"2024-03-11T22:48:01.544086+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:47:59.528950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:47:59.528950+0000","last_clean_scrub_stamp":"2024-03-11T22:47:59.528950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:28:37.501912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-11T22:48:00.547791+0000","last_change":"2024-03-11T22:48:00.546635+0000","last_active":"2024-03-11T22:48:00.547791+0000","last_peered":"2024-03-11T22:48:00.547791+0000","last_clean":"2024-03-11T22:48:00.547791+0000","last_became_active":"2024-03-11T22:48:00.546355+0000","last_became_peered":"2024-03-11T22:48:00.546355+0000","last_unstale":"2024-03-11T22:48:00.547791+0000","last_undegraded":"2024-03-11T22:48:00.547791+0000","last_fullsized":"2024-03-11T22:48:00.547791+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:47:59.528950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:47:59.528950+0000","last_clean_scrub_stamp":"2024-03-11T22:47:59.528950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:28:31.792354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-11T22:48:00.546943+0000","last_change":"2024-03-11T22:48:00.545535+0000","last_active":"2024-03-11T22:48:00.546943+0000","last_peered":"2024-03-11T22:48:00.546943+0000","last_clean":"2024-03-11T22:48:00.546943+0000","last_became_active":"2024-03-11T22:48:00.545254+0000","last_became_peered":"2024-03-11T22:48:00.545254+0000","last_unstale":"2024-03-11T22:48:00.546943+0000","last_undegraded":"2024-03-11T22:48:00.546943+0000","last_fullsized":"2024-03-11T22:48:00.546943+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:47:59.528950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:47:59.528950+0000","last_clean_scrub_stamp":"2024-03-11T22:47:59.528950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:14:17.934965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-11T22:48:00.548428+0000","last_change":"2024-03-11T22:48:00.546829+0000","last_active":"2024-03-11T22:48:00.548428+0000","last_peered":"2024-03-11T22:48:00.548428+0000","last_clean":"2024-03-11T22:48:00.548428+0000","last_became_active":"2024-03-11T22:48:00.546249+0000","last_became_peered":"2024-03-11T22:48:00.546249+0000","last_unstale":"2024-03-11T22:48:00.548428+0000","last_undegraded":"2024-03-11T22:48:00.548428+0000","last_fullsized":"2024-03-11T22:48:00.548428+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:47:59.528950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:47:59.528950+0000","last_clean_scrub_stamp":"2024-03-11T22:47:59.528950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:50:41.671462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":143,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-11T22:48:00.543532+0000","last_change":"2024-03-11T22:47:55.928747+0000","last_active":"2024-03-11T22:48:00.543532+0000","last_peered":"2024-03-11T22:48:00.543532+0000","last_clean":"2024-03-11T22:48:00.543532+0000","last_became_active":"2024-03-11T22:47:55.927708+0000","last_became_peered":"2024-03-11T22:47:55.927708+0000","last_unstale":"2024-03-11T22:48:00.543532+0000","last_undegraded":"2024-03-11T22:48:00.543532+0000","last_fullsized":"2024-03-11T22:48:00.543532+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:47:54.512460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:47:54.512460+0000","last_clean_scrub_stamp":"2024-03-11T22:47:54.512460+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:11:48.836881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27132,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344708,"statfs":{"total":96636764160,"available":96608980992,"internally_reserved":0,"allocated":114688,"data_stored":31355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:48:05.056 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-03-11T22:48:05.339 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:48:05.339 INFO:teuthology.orchestra.run.smithi132.stderr:dumped all 2024-03-11T22:48:05.351 INFO:teuthology.orchestra.run.smithi132.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-03-11T22:48:03.499335+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82556,"kb_used_data":1496,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032964,"statfs":{"total":289910292480,"available":289825755136,"internally_reserved":0,"allocated":1531904,"data_stored":1274801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000615"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-11T22:48:00.547691+0000","last_change":"2024-03-11T22:48:00.545536+0000","last_active":"2024-03-11T22:48:00.547691+0000","last_peered":"2024-03-11T22:48:00.547691+0000","last_clean":"2024-03-11T22:48:00.547691+0000","last_became_active":"2024-03-11T22:48:00.545246+0000","last_became_peered":"2024-03-11T22:48:00.545246+0000","last_unstale":"2024-03-11T22:48:00.547691+0000","last_undegraded":"2024-03-11T22:48:00.547691+0000","last_fullsized":"2024-03-11T22:48:00.547691+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:47:59.528950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:47:59.528950+0000","last_clean_scrub_stamp":"2024-03-11T22:47:59.528950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:14:24.117030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-11T22:48:00.548516+0000","last_change":"2024-03-11T22:48:00.547135+0000","last_active":"2024-03-11T22:48:00.548516+0000","last_peered":"2024-03-11T22:48:00.548516+0000","last_clean":"2024-03-11T22:48:00.548516+0000","last_became_active":"2024-03-11T22:48:00.546044+0000","last_became_peered":"2024-03-11T22:48:00.546044+0000","last_unstale":"2024-03-11T22:48:00.548516+0000","last_undegraded":"2024-03-11T22:48:00.548516+0000","last_fullsized":"2024-03-11T22:48:00.548516+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:47:59.528950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:47:59.528950+0000","last_clean_scrub_stamp":"2024-03-11T22:47:59.528950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:20:22.456780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-11T22:48:00.547762+0000","last_change":"2024-03-11T22:48:00.545568+0000","last_active":"2024-03-11T22:48:00.547762+0000","last_peered":"2024-03-11T22:48:00.547762+0000","last_clean":"2024-03-11T22:48:00.547762+0000","last_became_active":"2024-03-11T22:48:00.545283+0000","last_became_peered":"2024-03-11T22:48:00.545283+0000","last_unstale":"2024-03-11T22:48:00.547762+0000","last_undegraded":"2024-03-11T22:48:00.547762+0000","last_fullsized":"2024-03-11T22:48:00.547762+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:47:59.528950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:47:59.528950+0000","last_clean_scrub_stamp":"2024-03-11T22:47:59.528950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:05:32.693329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-11T22:48:00.548415+0000","last_change":"2024-03-11T22:48:00.546690+0000","last_active":"2024-03-11T22:48:00.548415+0000","last_peered":"2024-03-11T22:48:00.548415+0000","last_clean":"2024-03-11T22:48:00.548415+0000","last_became_active":"2024-03-11T22:48:00.546422+0000","last_became_peered":"2024-03-11T22:48:00.546422+0000","last_unstale":"2024-03-11T22:48:00.548415+0000","last_undegraded":"2024-03-11T22:48:00.548415+0000","last_fullsized":"2024-03-11T22:48:00.548415+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:47:59.528950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:47:59.528950+0000","last_clean_scrub_stamp":"2024-03-11T22:47:59.528950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:14:52.729483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-11T22:48:01.544086+0000","last_change":"2024-03-11T22:48:00.547841+0000","last_active":"2024-03-11T22:48:01.544086+0000","last_peered":"2024-03-11T22:48:01.544086+0000","last_clean":"2024-03-11T22:48:01.544086+0000","last_became_active":"2024-03-11T22:48:00.547567+0000","last_became_peered":"2024-03-11T22:48:00.547567+0000","last_unstale":"2024-03-11T22:48:01.544086+0000","last_undegraded":"2024-03-11T22:48:01.544086+0000","last_fullsized":"2024-03-11T22:48:01.544086+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:47:59.528950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:47:59.528950+0000","last_clean_scrub_stamp":"2024-03-11T22:47:59.528950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:28:37.501912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-11T22:48:00.547791+0000","last_change":"2024-03-11T22:48:00.546635+0000","last_active":"2024-03-11T22:48:00.547791+0000","last_peered":"2024-03-11T22:48:00.547791+0000","last_clean":"2024-03-11T22:48:00.547791+0000","last_became_active":"2024-03-11T22:48:00.546355+0000","last_became_peered":"2024-03-11T22:48:00.546355+0000","last_unstale":"2024-03-11T22:48:00.547791+0000","last_undegraded":"2024-03-11T22:48:00.547791+0000","last_fullsized":"2024-03-11T22:48:00.547791+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:47:59.528950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:47:59.528950+0000","last_clean_scrub_stamp":"2024-03-11T22:47:59.528950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:28:31.792354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-11T22:48:00.546943+0000","last_change":"2024-03-11T22:48:00.545535+0000","last_active":"2024-03-11T22:48:00.546943+0000","last_peered":"2024-03-11T22:48:00.546943+0000","last_clean":"2024-03-11T22:48:00.546943+0000","last_became_active":"2024-03-11T22:48:00.545254+0000","last_became_peered":"2024-03-11T22:48:00.545254+0000","last_unstale":"2024-03-11T22:48:00.546943+0000","last_undegraded":"2024-03-11T22:48:00.546943+0000","last_fullsized":"2024-03-11T22:48:00.546943+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:47:59.528950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:47:59.528950+0000","last_clean_scrub_stamp":"2024-03-11T22:47:59.528950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:14:17.934965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-11T22:48:00.548428+0000","last_change":"2024-03-11T22:48:00.546829+0000","last_active":"2024-03-11T22:48:00.548428+0000","last_peered":"2024-03-11T22:48:00.548428+0000","last_clean":"2024-03-11T22:48:00.548428+0000","last_became_active":"2024-03-11T22:48:00.546249+0000","last_became_peered":"2024-03-11T22:48:00.546249+0000","last_unstale":"2024-03-11T22:48:00.548428+0000","last_undegraded":"2024-03-11T22:48:00.548428+0000","last_fullsized":"2024-03-11T22:48:00.548428+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:47:59.528950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:47:59.528950+0000","last_clean_scrub_stamp":"2024-03-11T22:47:59.528950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:50:41.671462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":143,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-11T22:48:00.543532+0000","last_change":"2024-03-11T22:47:55.928747+0000","last_active":"2024-03-11T22:48:00.543532+0000","last_peered":"2024-03-11T22:48:00.543532+0000","last_clean":"2024-03-11T22:48:00.543532+0000","last_became_active":"2024-03-11T22:47:55.927708+0000","last_became_peered":"2024-03-11T22:47:55.927708+0000","last_unstale":"2024-03-11T22:48:00.543532+0000","last_undegraded":"2024-03-11T22:48:00.543532+0000","last_fullsized":"2024-03-11T22:48:00.543532+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:47:54.512460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:47:54.512460+0000","last_clean_scrub_stamp":"2024-03-11T22:47:54.512460+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:11:48.836881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27132,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344708,"statfs":{"total":96636764160,"available":96608980992,"internally_reserved":0,"allocated":114688,"data_stored":31355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:48:05.352 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-03-11T22:48:05.352 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2024-03-11T22:48:05.352 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2024-03-11T22:48:05.352 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2024-03-11T22:48:05.669 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-11T22:48:05.669 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-11T22:48:05.681 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2024-03-11T22:48:05.682 INFO:teuthology.run_tasks:Running task exec... 2024-03-11T22:48:05.692 INFO:teuthology.task.exec:Executing custom commands... 2024-03-11T22:48:05.692 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi132.front.sepia.ceph.com 2024-03-11T22:48:05.692 DEBUG:teuthology.orchestra.run.smithi132:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-03-11T22:48:06.058 INFO:teuthology.orchestra.run.smithi132.stdout:[client.mirror] 2024-03-11T22:48:06.059 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQDdie9lJ5nFMhAA+4V/28Td6xrE7+4tOe4JEw== 2024-03-11T22:48:06.059 INFO:teuthology.orchestra.run.smithi132.stdout: caps mon = "profile rbd-mirror-peer" 2024-03-11T22:48:06.059 INFO:teuthology.orchestra.run.smithi132.stdout: caps osd = "profile rbd" 2024-03-11T22:48:06.059 INFO:teuthology.orchestra.run.smithi132.stderr:updated caps for client.mirror 2024-03-11T22:48:06.072 DEBUG:teuthology.orchestra.run.smithi132:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-03-11T22:48:06.435 INFO:teuthology.orchestra.run.smithi132.stdout:[client.mirror.0] 2024-03-11T22:48:06.435 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQDdie9l17QzNhAA1SWbNnege6pXuycsc0npdw== 2024-03-11T22:48:06.435 INFO:teuthology.orchestra.run.smithi132.stdout: caps mon = "profile rbd-mirror" 2024-03-11T22:48:06.435 INFO:teuthology.orchestra.run.smithi132.stdout: caps osd = "profile rbd" 2024-03-11T22:48:06.435 INFO:teuthology.orchestra.run.smithi132.stderr:updated caps for client.mirror.0 2024-03-11T22:48:06.448 DEBUG:teuthology.orchestra.run.smithi132:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-03-11T22:48:06.811 INFO:teuthology.orchestra.run.smithi132.stdout:[client.mirror.1] 2024-03-11T22:48:06.811 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQDeie9lNr9jABAAE1afSLUktMTAX0tVN3wH2w== 2024-03-11T22:48:06.811 INFO:teuthology.orchestra.run.smithi132.stdout: caps mon = "profile rbd-mirror" 2024-03-11T22:48:06.812 INFO:teuthology.orchestra.run.smithi132.stdout: caps osd = "profile rbd" 2024-03-11T22:48:06.812 INFO:teuthology.orchestra.run.smithi132.stderr:updated caps for client.mirror.1 2024-03-11T22:48:06.825 DEBUG:teuthology.orchestra.run.smithi132:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-03-11T22:48:07.187 INFO:teuthology.orchestra.run.smithi132.stdout:[client.mirror.2] 2024-03-11T22:48:07.188 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQDeie9liOw6BhAAKbMgwcmyT9Etrco1JFn0dQ== 2024-03-11T22:48:07.188 INFO:teuthology.orchestra.run.smithi132.stdout: caps mon = "profile rbd-mirror" 2024-03-11T22:48:07.188 INFO:teuthology.orchestra.run.smithi132.stdout: caps osd = "profile rbd" 2024-03-11T22:48:07.188 INFO:teuthology.orchestra.run.smithi132.stderr:updated caps for client.mirror.2 2024-03-11T22:48:07.201 DEBUG:teuthology.orchestra.run.smithi132:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-03-11T22:48:07.564 INFO:teuthology.orchestra.run.smithi132.stdout:[client.mirror.3] 2024-03-11T22:48:07.564 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQDeie9lA8oPDBAAdo3fTzAGL+ODxMwEqUBHEg== 2024-03-11T22:48:07.564 INFO:teuthology.orchestra.run.smithi132.stdout: caps mon = "profile rbd-mirror" 2024-03-11T22:48:07.564 INFO:teuthology.orchestra.run.smithi132.stdout: caps osd = "profile rbd" 2024-03-11T22:48:07.564 INFO:teuthology.orchestra.run.smithi132.stderr:updated caps for client.mirror.3 2024-03-11T22:48:07.577 DEBUG:teuthology.orchestra.run.smithi132:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-03-11T22:48:07.938 INFO:teuthology.orchestra.run.smithi132.stdout:[client.mirror] 2024-03-11T22:48:07.938 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQACiu9lNtLYDhAAszjbT/to6zICOSn/nO3uFA== 2024-03-11T22:48:07.938 INFO:teuthology.orchestra.run.smithi132.stdout: caps mon = "profile rbd-mirror-peer" 2024-03-11T22:48:07.938 INFO:teuthology.orchestra.run.smithi132.stdout: caps osd = "profile rbd" 2024-03-11T22:48:07.938 INFO:teuthology.orchestra.run.smithi132.stderr:updated caps for client.mirror 2024-03-11T22:48:07.951 DEBUG:teuthology.orchestra.run.smithi132:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-03-11T22:48:08.312 INFO:teuthology.orchestra.run.smithi132.stdout:[client.mirror.0] 2024-03-11T22:48:08.312 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQACiu9l9VyrFBAAQkuCTDjv/iLToWza0ePB5w== 2024-03-11T22:48:08.313 INFO:teuthology.orchestra.run.smithi132.stdout: caps mon = "profile rbd-mirror" 2024-03-11T22:48:08.313 INFO:teuthology.orchestra.run.smithi132.stdout: caps osd = "profile rbd" 2024-03-11T22:48:08.313 INFO:teuthology.orchestra.run.smithi132.stderr:updated caps for client.mirror.0 2024-03-11T22:48:08.327 DEBUG:teuthology.orchestra.run.smithi132:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-03-11T22:48:08.691 INFO:teuthology.orchestra.run.smithi132.stdout:[client.mirror.1] 2024-03-11T22:48:08.691 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQACiu9l9W1/GhAAxNc3mhPRr7Q/LtK4nzxbdQ== 2024-03-11T22:48:08.691 INFO:teuthology.orchestra.run.smithi132.stdout: caps mon = "profile rbd-mirror" 2024-03-11T22:48:08.691 INFO:teuthology.orchestra.run.smithi132.stdout: caps osd = "profile rbd" 2024-03-11T22:48:08.691 INFO:teuthology.orchestra.run.smithi132.stderr:updated caps for client.mirror.1 2024-03-11T22:48:08.704 DEBUG:teuthology.orchestra.run.smithi132:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-03-11T22:48:09.069 INFO:teuthology.orchestra.run.smithi132.stdout:[client.mirror.2] 2024-03-11T22:48:09.070 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQACiu9lwmRVIBAAKfW08bQoLBKly5+7lfBrrQ== 2024-03-11T22:48:09.070 INFO:teuthology.orchestra.run.smithi132.stdout: caps mon = "profile rbd-mirror" 2024-03-11T22:48:09.070 INFO:teuthology.orchestra.run.smithi132.stdout: caps osd = "profile rbd" 2024-03-11T22:48:09.070 INFO:teuthology.orchestra.run.smithi132.stderr:updated caps for client.mirror.2 2024-03-11T22:48:09.083 DEBUG:teuthology.orchestra.run.smithi132:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-03-11T22:48:09.447 INFO:teuthology.orchestra.run.smithi132.stdout:[client.mirror.3] 2024-03-11T22:48:09.447 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQACiu9lAsAbJhAAoVbqNNiPS8wYxf6YXjQ+Pg== 2024-03-11T22:48:09.447 INFO:teuthology.orchestra.run.smithi132.stdout: caps mon = "profile rbd-mirror" 2024-03-11T22:48:09.447 INFO:teuthology.orchestra.run.smithi132.stdout: caps osd = "profile rbd" 2024-03-11T22:48:09.448 INFO:teuthology.orchestra.run.smithi132.stderr:updated caps for client.mirror.3 2024-03-11T22:48:09.461 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-03-11T22:48:09.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-03-11T22:48:09.473 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-03-11T22:48:09.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-03-11T22:48:09.476 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-03-11T22:48:09.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-03-11T22:48:09.488 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-03-11T22:48:09.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-03-11T22:48:09.490 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-03-11T22:48:09.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-03-11T22:48:09.505 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-03-11T22:48:09.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-03-11T22:48:09.507 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-03-11T22:48:09.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-03-11T22:48:09.522 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-03-11T22:48:09.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-03-11T22:48:09.524 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-03-11T22:48:09.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-03-11T22:48:09.538 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-03-11T22:48:09.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-03-11T22:48:09.540 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-03-11T22:48:09.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-03-11T22:48:09.553 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-03-11T22:48:09.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-03-11T22:48:09.555 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-03-11T22:48:09.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-03-11T22:48:09.568 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-03-11T22:48:09.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-03-11T22:48:09.571 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-03-11T22:48:09.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-03-11T22:48:09.583 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-03-11T22:48:09.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-03-11T22:48:09.586 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-03-11T22:48:09.602 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1710197289 2024-03-11T22:48:09.602 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-03-11T22:48:09.602 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-03-11T22:48:09.617 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1710197289 2024-03-11T22:48:09.617 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-03-11T22:48:09.617 INFO:teuthology.run_tasks:Running task workunit... 2024-03-11T22:48:09.646 INFO:tasks.workunit:Pulling workunits from ref a4679433626e24f222f64f757c6ab31760167603 2024-03-11T22:48:09.647 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-11T22:48:09.647 DEBUG:teuthology.orchestra.run.smithi132:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-03-11T22:48:09.648 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2024-03-11T22:48:09.648 INFO:tasks.rbd_mirror_thrash:waiting for 108 secs before thrashing 2024-03-11T22:48:09.648 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2024-03-11T22:48:09.648 INFO:tasks.rbd_mirror_thrash:waiting for 69 secs before thrashing 2024-03-11T22:48:09.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:48:09.662 INFO:teuthology.orchestra.run.smithi132.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2024-03-11T22:48:09.663 DEBUG:teuthology.orchestra.run.smithi132:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-03-11T22:48:09.717 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-03-11T22:48:09.718 DEBUG:teuthology.orchestra.run.smithi132:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2024-03-11T22:48:09.772 INFO:tasks.workunit:timeout=6h 2024-03-11T22:48:09.773 INFO:tasks.workunit:cleanup=True 2024-03-11T22:48:09.773 DEBUG:teuthology.orchestra.run.smithi132:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://github.com/idryomov/ceph /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout a4679433626e24f222f64f757c6ab31760167603 2024-03-11T22:48:09.828 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2024-03-11T22:49:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Note: switching to 'a4679433626e24f222f64f757c6ab31760167603'. 2024-03-11T22:49:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:49:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-11T22:49:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-11T22:49:05.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:state without impacting any branches by switching back to a branch. 2024-03-11T22:49:05.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:49:05.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-11T22:49:05.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-11T22:49:05.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:49:05.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: git switch -c 2024-03-11T22:49:05.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:49:05.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Or undo this operation with: 2024-03-11T22:49:05.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:49:05.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: git switch - 2024-03-11T22:49:05.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:49:05.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-11T22:49:05.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:49:05.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:HEAD is now at a4679433626 qa/workunits/rbd: switch rbd-mirror workunits to bash 2024-03-11T22:49:05.353 DEBUG:teuthology.orchestra.run.smithi132:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2024-03-11T22:49:05.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-11T22:49:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-03-11T22:49:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-11T22:49:06.346 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-11T22:49:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-11T22:49:06.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-03-11T22:49:06.459 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-03-11T22:49:06.459 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-11T22:49:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-03-11T22:49:06.502 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-11T22:49:06.502 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2024-03-11T22:49:06.517 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2024-03-11T22:49:06.519 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2024-03-11T22:49:06.519 DEBUG:teuthology.orchestra.run.smithi132:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a4679433626e24f222f64f757c6ab31760167603 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-03-11T22:49:06.576 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ IMAGE_COUNT=50 2024-03-11T22:49:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ export LOCKDEP=0 2024-03-11T22:49:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ LOCKDEP=0 2024-03-11T22:49:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-03-11T22:49:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2024-03-11T22:49:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ type xmlstarlet 2024-03-11T22:49:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ XMLSTARLET=xmlstarlet 2024-03-11T22:49:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ RBD_MIRROR_INSTANCES=4 2024-03-11T22:49:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CLUSTER1=cluster1 2024-03-11T22:49:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CLUSTER2=cluster2 2024-03-11T22:49:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ PEER_CLUSTER_SUFFIX= 2024-03-11T22:49:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ POOL=mirror 2024-03-11T22:49:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ PARENT_POOL=mirror_parent 2024-03-11T22:49:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ NS1=ns1 2024-03-11T22:49:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ NS2=ns2 2024-03-11T22:49:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ TEMPDIR= 2024-03-11T22:49:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ID=mirror 2024-03-11T22:49:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ RBD_IMAGE_FEATURES=layering 2024-03-11T22:49:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2024-03-11T22:49:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ RBD_MIRROR_MODE=snapshot 2024-03-11T22:49:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ MIRROR_POOL_MODE=image 2024-03-11T22:49:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' snapshot = snapshot ']' 2024-03-11T22:49:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ MIRROR_POOL_MODE=image 2024-03-11T22:49:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ export 'CEPH_ARGS=--id mirror' 2024-03-11T22:49:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS='--id mirror' 2024-03-11T22:49:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ LAST_MIRROR_INSTANCE=3 2024-03-11T22:49:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-03-11T22:49:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2024-03-11T22:49:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2024-03-11T22:49:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_BIN=. 2024-03-11T22:49:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_SRC=. 2024-03-11T22:49:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' -e CMakeCache.txt ']' 2024-03-11T22:49:06.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 -gt 0 ']' 2024-03-11T22:49:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ setup 2024-03-11T22:49:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local c 2024-03-11T22:49:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ trap 'cleanup $?' INT TERM EXIT 2024-03-11T22:49:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ setup_tempdir 2024-03-11T22:49:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T22:49:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mktemp -d 2024-03-11T22:49:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ TEMPDIR=/tmp/tmp.7mIVT0l9FS 2024-03-11T22:49:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -z 1 ']' 2024-03-11T22:49:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ setup_pools cluster1 cluster2 2024-03-11T22:49:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:49:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local remote_cluster=cluster2 2024-03-11T22:49:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mon_map_file 2024-03-11T22:49:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mon_addr 2024-03-11T22:49:06.584 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local admin_key_file 2024-03-11T22:49:06.584 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local uuid 2024-03-11T22:49:06.584 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ CEPH_ARGS= 2024-03-11T22:49:06.584 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2024-03-11T22:49:07.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:pool 'mirror' already exists 2024-03-11T22:49:07.766 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ CEPH_ARGS= 2024-03-11T22:49:07.766 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2024-03-11T22:49:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:pool 'mirror_parent' already exists 2024-03-11T22:49:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ CEPH_ARGS= 2024-03-11T22:49:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 pool init mirror 2024-03-11T22:49:11.691 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ CEPH_ARGS= 2024-03-11T22:49:11.691 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2024-03-11T22:49:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T22:49:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ CEPH_ARGS= 2024-03-11T22:49:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2024-03-11T22:49:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2024-03-11T22:49:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2024-03-11T22:49:14.819 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2024-03-11T22:49:14.853 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2024-03-11T22:49:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2024-03-11T22:49:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -z ']' 2024-03-11T22:49:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -z ']' 2024-03-11T22:49:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ peer_add cluster1 mirror cluster2 2024-03-11T22:49:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:49:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:49:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:49:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:49:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local client_cluster=cluster2 2024-03-11T22:49:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:49:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local remote_cluster=cluster2 2024-03-11T22:49:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local uuid_var_name 2024-03-11T22:49:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T22:49:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local error_code 2024-03-11T22:49:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local peer_uuid 2024-03-11T22:49:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 16 32 2024-03-11T22:49:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T22:49:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2024-03-11T22:49:14.967 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ peer_uuid=a05ccac1-a6e1-41f1-b76d-377fa37a2621 2024-03-11T22:49:14.967 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=0 2024-03-11T22:49:14.967 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T22:49:14.967 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 0 -eq 17 ']' 2024-03-11T22:49:14.967 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 0 -eq 0 2024-03-11T22:49:14.967 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T22:49:14.967 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:49:14.967 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ peer_add cluster1 mirror_parent cluster2 2024-03-11T22:49:14.967 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:49:14.967 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:49:14.967 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror_parent 2024-03-11T22:49:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:49:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local client_cluster=cluster2 2024-03-11T22:49:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:49:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local remote_cluster=cluster2 2024-03-11T22:49:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local uuid_var_name 2024-03-11T22:49:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T22:49:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local error_code 2024-03-11T22:49:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local peer_uuid 2024-03-11T22:49:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 16 32 2024-03-11T22:49:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T22:49:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2024-03-11T22:49:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ peer_uuid=1b3e982d-052e-4b35-ad6d-2c8ec02599b7 2024-03-11T22:49:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=0 2024-03-11T22:49:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T22:49:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 0 -eq 17 ']' 2024-03-11T22:49:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 0 -eq 0 2024-03-11T22:49:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T22:49:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:49:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ setup_pools cluster2 cluster1 2024-03-11T22:49:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:49:15.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local remote_cluster=cluster1 2024-03-11T22:49:15.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mon_map_file 2024-03-11T22:49:15.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mon_addr 2024-03-11T22:49:15.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local admin_key_file 2024-03-11T22:49:15.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local uuid 2024-03-11T22:49:15.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ CEPH_ARGS= 2024-03-11T22:49:15.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2024-03-11T22:49:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:pool 'mirror' already exists 2024-03-11T22:49:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ CEPH_ARGS= 2024-03-11T22:49:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2024-03-11T22:49:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:pool 'mirror_parent' already exists 2024-03-11T22:49:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ CEPH_ARGS= 2024-03-11T22:49:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 pool init mirror 2024-03-11T22:49:18.649 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8940036952718868) > weight (0.25) 2024-03-11T22:49:18.649 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7809924711506288) > weight (0.25) 2024-03-11T22:49:18.649 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.2714151765319611) > weight (0.25) 2024-03-11T22:49:18.649 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.3291790602932526) > weight (0.25) 2024-03-11T22:49:18.649 INFO:tasks.rbd_mirror_thrash:waiting for 90 secs before thrashing 2024-03-11T22:49:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ CEPH_ARGS= 2024-03-11T22:49:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2024-03-11T22:49:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T22:49:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ CEPH_ARGS= 2024-03-11T22:49:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2024-03-11T22:49:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2024-03-11T22:49:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2024-03-11T22:49:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2024-03-11T22:49:22.799 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2024-03-11T22:49:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2024-03-11T22:49:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -z ']' 2024-03-11T22:49:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -z ']' 2024-03-11T22:49:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ peer_add cluster2 mirror cluster1 2024-03-11T22:49:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:49:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:49:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:49:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:49:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local client_cluster=cluster1 2024-03-11T22:49:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:49:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local remote_cluster=cluster1 2024-03-11T22:49:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local uuid_var_name 2024-03-11T22:49:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T22:49:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local error_code 2024-03-11T22:49:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local peer_uuid 2024-03-11T22:49:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 16 32 2024-03-11T22:49:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T22:49:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2024-03-11T22:49:22.910 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ peer_uuid=66ab8acb-3427-481b-81cd-aecdb6734b3f 2024-03-11T22:49:22.911 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=0 2024-03-11T22:49:22.911 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T22:49:22.911 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 0 -eq 17 ']' 2024-03-11T22:49:22.911 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 0 -eq 0 2024-03-11T22:49:22.911 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T22:49:22.911 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:49:22.911 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ peer_add cluster2 mirror_parent cluster1 2024-03-11T22:49:22.911 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:49:22.911 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:49:22.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror_parent 2024-03-11T22:49:22.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:49:22.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local client_cluster=cluster1 2024-03-11T22:49:22.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:49:22.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local remote_cluster=cluster1 2024-03-11T22:49:22.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local uuid_var_name 2024-03-11T22:49:22.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T22:49:22.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local error_code 2024-03-11T22:49:22.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local peer_uuid 2024-03-11T22:49:22.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 16 32 2024-03-11T22:49:22.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T22:49:22.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2024-03-11T22:49:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ peer_uuid=b37f8a0b-8eaa-42f8-b686-569b38ed08cd 2024-03-11T22:49:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=0 2024-03-11T22:49:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T22:49:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 0 -eq 17 ']' 2024-03-11T22:49:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 0 -eq 0 2024-03-11T22:49:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T22:49:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:49:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T22:49:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ start_mirrors cluster1 2024-03-11T22:49:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:49:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ seq 0 3 2024-03-11T22:49:23.037 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:49:23.037 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ start_mirror cluster1:0 2024-03-11T22:49:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1:0 2024-03-11T22:49:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local instance 2024-03-11T22:49:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-11T22:49:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local val=cluster1:0 2024-03-11T22:49:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster_var_name=cluster 2024-03-11T22:49:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local instance_var_name=instance 2024-03-11T22:49:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cluster=cluster1 2024-03-11T22:49:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ instance=0 2024-03-11T22:49:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-11T22:49:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ eval cluster=cluster1 2024-03-11T22:49:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:49:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ eval instance=0 2024-03-11T22:49:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T22:49:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n 1 2024-03-11T22:49:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 2024-03-11T22:49:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:49:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ start_mirror cluster1:1 2024-03-11T22:49:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1:1 2024-03-11T22:49:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local instance 2024-03-11T22:49:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-11T22:49:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local val=cluster1:1 2024-03-11T22:49:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster_var_name=cluster 2024-03-11T22:49:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local instance_var_name=instance 2024-03-11T22:49:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cluster=cluster1 2024-03-11T22:49:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ instance=1 2024-03-11T22:49:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-11T22:49:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ eval cluster=cluster1 2024-03-11T22:49:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:49:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ eval instance=1 2024-03-11T22:49:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T22:49:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n 1 2024-03-11T22:49:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 2024-03-11T22:49:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:49:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ start_mirror cluster1:2 2024-03-11T22:49:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1:2 2024-03-11T22:49:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local instance 2024-03-11T22:49:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-03-11T22:49:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local val=cluster1:2 2024-03-11T22:49:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster_var_name=cluster 2024-03-11T22:49:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local instance_var_name=instance 2024-03-11T22:49:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cluster=cluster1 2024-03-11T22:49:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ instance=2 2024-03-11T22:49:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 2 = cluster1:2 ']' 2024-03-11T22:49:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ eval cluster=cluster1 2024-03-11T22:49:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:49:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ eval instance=2 2024-03-11T22:49:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T22:49:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n 1 2024-03-11T22:49:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 2024-03-11T22:49:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:49:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ start_mirror cluster1:3 2024-03-11T22:49:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1:3 2024-03-11T22:49:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local instance 2024-03-11T22:49:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-03-11T22:49:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local val=cluster1:3 2024-03-11T22:49:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster_var_name=cluster 2024-03-11T22:49:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local instance_var_name=instance 2024-03-11T22:49:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cluster=cluster1 2024-03-11T22:49:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ instance=3 2024-03-11T22:49:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 3 = cluster1:3 ']' 2024-03-11T22:49:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ eval cluster=cluster1 2024-03-11T22:49:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:49:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ eval instance=3 2024-03-11T22:49:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=3 2024-03-11T22:49:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n 1 2024-03-11T22:49:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 2024-03-11T22:49:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ start_mirrors cluster2 2024-03-11T22:49:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:49:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ seq 0 3 2024-03-11T22:49:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:49:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ start_mirror cluster2:0 2024-03-11T22:49:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2:0 2024-03-11T22:49:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local instance 2024-03-11T22:49:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-11T22:49:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local val=cluster2:0 2024-03-11T22:49:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster_var_name=cluster 2024-03-11T22:49:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local instance_var_name=instance 2024-03-11T22:49:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cluster=cluster2 2024-03-11T22:49:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ instance=0 2024-03-11T22:49:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-11T22:49:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ eval cluster=cluster2 2024-03-11T22:49:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster2 2024-03-11T22:49:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ eval instance=0 2024-03-11T22:49:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T22:49:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n 1 2024-03-11T22:49:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 2024-03-11T22:49:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:49:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ start_mirror cluster2:1 2024-03-11T22:49:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2:1 2024-03-11T22:49:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local instance 2024-03-11T22:49:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-03-11T22:49:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local val=cluster2:1 2024-03-11T22:49:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster_var_name=cluster 2024-03-11T22:49:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local instance_var_name=instance 2024-03-11T22:49:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cluster=cluster2 2024-03-11T22:49:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ instance=1 2024-03-11T22:49:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 1 = cluster2:1 ']' 2024-03-11T22:49:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ eval cluster=cluster2 2024-03-11T22:49:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster2 2024-03-11T22:49:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ eval instance=1 2024-03-11T22:49:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T22:49:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n 1 2024-03-11T22:49:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 2024-03-11T22:49:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:49:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ start_mirror cluster2:2 2024-03-11T22:49:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2:2 2024-03-11T22:49:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local instance 2024-03-11T22:49:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set_cluster_instance cluster2:2 cluster instance 2024-03-11T22:49:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local val=cluster2:2 2024-03-11T22:49:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster_var_name=cluster 2024-03-11T22:49:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local instance_var_name=instance 2024-03-11T22:49:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cluster=cluster2 2024-03-11T22:49:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ instance=2 2024-03-11T22:49:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 2 = cluster2:2 ']' 2024-03-11T22:49:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ eval cluster=cluster2 2024-03-11T22:49:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster2 2024-03-11T22:49:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ eval instance=2 2024-03-11T22:49:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T22:49:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n 1 2024-03-11T22:49:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 2024-03-11T22:49:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:49:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ start_mirror cluster2:3 2024-03-11T22:49:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2:3 2024-03-11T22:49:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local instance 2024-03-11T22:49:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set_cluster_instance cluster2:3 cluster instance 2024-03-11T22:49:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local val=cluster2:3 2024-03-11T22:49:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster_var_name=cluster 2024-03-11T22:49:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local instance_var_name=instance 2024-03-11T22:49:23.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cluster=cluster2 2024-03-11T22:49:23.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ instance=3 2024-03-11T22:49:23.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 3 = cluster2:3 ']' 2024-03-11T22:49:23.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ eval cluster=cluster2 2024-03-11T22:49:23.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster2 2024-03-11T22:49:23.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ eval instance=3 2024-03-11T22:49:23.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=3 2024-03-11T22:49:23.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n 1 2024-03-11T22:49:23.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 2024-03-11T22:49:23.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2024-03-11T22:49:23.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ tee -a /tmp/tmp.7mIVT0l9FS/rbd-mirror.test.log 2024-03-11T22:49:23.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ date '+%F %T' 2024-03-11T22:49:23.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo 2024-03-11 22:49:23 TEST: add image and test replay after client crashes 2024-03-11T22:49:23.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:2024-03-11 22:49:23 TEST: add image and test replay after client crashes 2024-03-11T22:49:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=test 2024-03-11T22:49:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 512M 2024-03-11T22:49:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:49:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:49:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:49:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:49:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:49:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:49:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T22:49:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T22:49:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:49:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror test 512M 2024-03-11T22:49:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:49:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:49:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:49:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:49:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:49:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:49:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T22:49:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 512M ']' 2024-03-11T22:49:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=512M 2024-03-11T22:49:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:49:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/test 2024-03-11T22:49:23.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T22:49:23.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:49:23.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T22:49:23.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:49:23.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror test snapshot 2024-03-11T22:49:23.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:49:23.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:49:23.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:49:23.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T22:49:23.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2024-03-11T22:49:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T22:49:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/test 2024-03-11T22:49:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'test': 2024-03-11T22:49:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 512 MiB in 128 objects 2024-03-11T22:49:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T22:49:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T22:49:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 10a98d5329d0 2024-03-11T22:49:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.10a98d5329d0 2024-03-11T22:49:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T22:49:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T22:49:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T22:49:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T22:49:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 22:49:23 2024 2024-03-11T22:49:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 22:49:23 2024 2024-03-11T22:49:23.746 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 22:49:23 2024 2024-03-11T22:49:23.746 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T22:49:23.746 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T22:49:23.746 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: d6055756-c87a-420f-9f23-02c2bbaa6857 2024-03-11T22:49:23.746 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T22:49:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-11T22:49:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:49:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:49:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:49:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-11T22:49:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:49:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:49:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:49:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T22:49:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T22:49:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:49:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T22:49:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-11T22:49:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:49:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:49:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:49:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T22:49:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T22:49:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T22:49:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-11T22:49:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T22:49:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T22:49:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:49:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T22:49:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T22:49:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T22:49:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:49:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T22:49:24.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T22:49:24.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T22:49:24.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:49:24.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-11T22:49:24.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T22:49:24.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:49:24.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:49:24.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T22:49:24.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T22:49:24.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:49:24.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:49:24.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:49:24.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T22:49:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T22:49:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:49:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T22:49:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:49:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T22:49:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T22:49:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:49:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:49:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T22:49:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T22:49:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:49:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:49:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:49:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T22:49:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:49:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:49:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T22:49:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T22:49:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T22:49:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:49:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:49:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:49:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-11T22:49:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-11T22:49:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T22:49:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T22:49:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:49:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-03-11T22:49:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T22:49:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:49:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:49:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T22:49:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T22:49:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:49:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:49:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:49:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T22:49:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T22:49:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:49:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T22:49:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T22:49:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T22:49:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T22:49:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:49:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:49:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T22:49:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T22:49:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:49:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:49:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T22:49:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T22:49:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:49:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:49:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T22:49:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T22:49:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T22:49:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T22:49:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T22:49:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T22:49:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-03-11T22:49:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-11T22:49:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T22:49:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T22:49:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:49:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-03-11T22:49:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T22:49:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:49:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:49:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T22:49:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T22:49:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:49:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:49:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:49:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T22:49:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T22:49:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:49:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T22:49:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T22:49:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T22:49:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T22:49:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:49:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:49:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T22:49:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T22:49:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:49:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:49:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T22:49:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T22:49:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:49:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:49:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T22:49:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T22:49:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T22:49:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T22:49:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T22:49:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T22:49:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-03-11T22:49:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-11T22:49:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T22:49:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T22:49:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:49:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-03-11T22:49:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:3 2024-03-11T22:49:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:49:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:49:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-03-11T22:49:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:3 2024-03-11T22:49:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:49:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:49:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:49:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=3 2024-03-11T22:49:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 3 = cluster1:3 ']' 2024-03-11T22:49:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:49:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=3 2024-03-11T22:49:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T22:49:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-03-11T22:49:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:3 2024-03-11T22:49:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:49:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:49:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-03-11T22:49:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:3 2024-03-11T22:49:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:49:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:49:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T22:49:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 3 = cluster1:3 ']' 2024-03-11T22:49:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:49:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:49:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=3 2024-03-11T22:49:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=3 2024-03-11T22:49:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-03-11T22:49:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T22:49:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T22:49:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T22:49:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-03-11T22:49:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-11T22:49:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T22:49:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T22:49:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:49:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 1 2024-03-11T22:49:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T22:49:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T22:49:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:49:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-11T22:49:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T22:49:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:49:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:49:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T22:49:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T22:49:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:49:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:49:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:49:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T22:49:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T22:49:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:49:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T22:49:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:49:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T22:49:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T22:49:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:49:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:49:26.134 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T22:49:26.134 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T22:49:26.134 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:49:26.134 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:49:26.134 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:26.134 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:49:26.134 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T22:49:26.134 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:49:26.134 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:49:26.134 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T22:49:26.134 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T22:49:26.134 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T22:49:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:49:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:49:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:49:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-11T22:49:26.230 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-11T22:49:26.230 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T22:49:26.230 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T22:49:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:49:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-03-11T22:49:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T22:49:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:49:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:49:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T22:49:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T22:49:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:49:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:49:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:49:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T22:49:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T22:49:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:49:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T22:49:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T22:49:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T22:49:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T22:49:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:49:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:49:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T22:49:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T22:49:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:49:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:49:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T22:49:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T22:49:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:49:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:49:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T22:49:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T22:49:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T22:49:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T22:49:26.252 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T22:49:26.252 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T22:49:26.252 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-03-11T22:49:26.323 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-11T22:49:26.323 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T22:49:26.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T22:49:26.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:49:26.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-03-11T22:49:26.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T22:49:26.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:49:26.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:49:26.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T22:49:26.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T22:49:26.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:49:26.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:49:26.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:49:26.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T22:49:26.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T22:49:26.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:49:26.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:26.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T22:49:26.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T22:49:26.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T22:49:26.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T22:49:26.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:49:26.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:49:26.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T22:49:26.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T22:49:26.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:49:26.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:49:26.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:26.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T22:49:26.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T22:49:26.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:49:26.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:49:26.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T22:49:26.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T22:49:26.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T22:49:26.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T22:49:26.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T22:49:26.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T22:49:26.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-03-11T22:49:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-11T22:49:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T22:49:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T22:49:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:49:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-03-11T22:49:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:3 2024-03-11T22:49:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:49:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:49:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-03-11T22:49:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:3 2024-03-11T22:49:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:49:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:49:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:49:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=3 2024-03-11T22:49:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 3 = cluster1:3 ']' 2024-03-11T22:49:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:49:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=3 2024-03-11T22:49:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T22:49:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-03-11T22:49:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:3 2024-03-11T22:49:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:49:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:49:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-03-11T22:49:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:3 2024-03-11T22:49:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:49:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:49:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T22:49:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 3 = cluster1:3 ']' 2024-03-11T22:49:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:49:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:49:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=3 2024-03-11T22:49:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=3 2024-03-11T22:49:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-03-11T22:49:26.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T22:49:26.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T22:49:26.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T22:49:26.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-03-11T22:49:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-11T22:49:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T22:49:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T22:49:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:49:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 2 2024-03-11T22:49:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T22:49:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T22:49:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:49:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-11T22:49:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T22:49:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:49:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:49:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T22:49:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T22:49:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:49:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:49:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:49:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T22:49:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T22:49:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:49:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T22:49:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:49:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T22:49:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T22:49:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:49:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:49:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T22:49:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T22:49:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:49:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:49:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:49:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T22:49:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:49:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:49:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T22:49:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T22:49:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T22:49:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:49:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:49:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:49:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-11T22:49:28.610 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-11T22:49:28.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T22:49:28.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T22:49:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:49:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-03-11T22:49:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T22:49:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:49:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:49:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T22:49:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T22:49:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:49:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:49:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:49:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T22:49:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T22:49:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:49:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T22:49:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T22:49:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T22:49:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T22:49:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:49:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:49:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T22:49:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T22:49:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:49:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:49:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T22:49:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T22:49:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:49:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:49:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T22:49:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T22:49:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T22:49:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T22:49:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T22:49:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T22:49:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-03-11T22:49:28.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-11T22:49:28.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T22:49:28.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T22:49:28.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:49:28.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-03-11T22:49:28.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T22:49:28.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:49:28.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:49:28.710 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T22:49:28.710 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T22:49:28.710 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:49:28.710 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:49:28.710 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:49:28.710 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T22:49:28.710 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T22:49:28.710 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:49:28.711 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:28.711 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T22:49:28.711 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T22:49:28.711 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T22:49:28.711 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T22:49:28.711 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:49:28.711 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:49:28.711 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T22:49:28.711 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T22:49:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:49:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:49:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T22:49:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T22:49:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:49:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:49:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T22:49:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T22:49:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T22:49:28.725 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T22:49:28.725 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T22:49:28.725 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T22:49:28.725 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-03-11T22:49:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-11T22:49:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T22:49:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T22:49:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:49:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-03-11T22:49:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:3 2024-03-11T22:49:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:49:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:49:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-03-11T22:49:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:3 2024-03-11T22:49:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:49:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:49:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:49:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=3 2024-03-11T22:49:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 3 = cluster1:3 ']' 2024-03-11T22:49:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:49:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=3 2024-03-11T22:49:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T22:49:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-03-11T22:49:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:3 2024-03-11T22:49:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:49:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:49:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-03-11T22:49:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:3 2024-03-11T22:49:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:49:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:49:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T22:49:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 3 = cluster1:3 ']' 2024-03-11T22:49:28.802 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:49:28.802 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:49:28.802 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=3 2024-03-11T22:49:28.802 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=3 2024-03-11T22:49:28.802 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-03-11T22:49:28.816 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T22:49:28.816 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T22:49:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T22:49:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-03-11T22:49:28.887 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-11T22:49:28.887 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T22:49:28.887 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T22:49:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:49:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 4 2024-03-11T22:49:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T22:49:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T22:49:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:49:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-11T22:49:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T22:49:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:49:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:49:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T22:49:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T22:49:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:49:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:49:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:49:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T22:49:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T22:49:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:49:32.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:32.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T22:49:32.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:49:32.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T22:49:32.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T22:49:32.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:49:32.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:49:32.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T22:49:32.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T22:49:32.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:49:32.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:49:32.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:32.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:49:32.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T22:49:32.895 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:49:32.895 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:49:32.895 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T22:49:32.895 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T22:49:32.895 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T22:49:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:49:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:49:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:49:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-11T22:49:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-11T22:49:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T22:49:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T22:49:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:49:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-03-11T22:49:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T22:49:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:49:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:49:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T22:49:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T22:49:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:49:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:49:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:49:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T22:49:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T22:49:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:49:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T22:49:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T22:49:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T22:49:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T22:49:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:49:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:49:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T22:49:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T22:49:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:49:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:49:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T22:49:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T22:49:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:49:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:49:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T22:49:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T22:49:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T22:49:33.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T22:49:33.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T22:49:33.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T22:49:33.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-03-11T22:49:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-11T22:49:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T22:49:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T22:49:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:49:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-03-11T22:49:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T22:49:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:49:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:49:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T22:49:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T22:49:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:49:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:49:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:49:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T22:49:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T22:49:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:49:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T22:49:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T22:49:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T22:49:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T22:49:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:49:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:49:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T22:49:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T22:49:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:49:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:49:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T22:49:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T22:49:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:49:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:49:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T22:49:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T22:49:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T22:49:33.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T22:49:33.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T22:49:33.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T22:49:33.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-03-11T22:49:33.176 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-11T22:49:33.176 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T22:49:33.176 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T22:49:33.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:49:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-03-11T22:49:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:3 2024-03-11T22:49:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:49:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:49:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-03-11T22:49:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:3 2024-03-11T22:49:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:49:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:49:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:49:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=3 2024-03-11T22:49:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 3 = cluster1:3 ']' 2024-03-11T22:49:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:49:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=3 2024-03-11T22:49:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T22:49:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-03-11T22:49:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:3 2024-03-11T22:49:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:49:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:49:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-03-11T22:49:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:3 2024-03-11T22:49:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:49:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:49:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T22:49:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 3 = cluster1:3 ']' 2024-03-11T22:49:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:49:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:49:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=3 2024-03-11T22:49:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=3 2024-03-11T22:49:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-03-11T22:49:33.198 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T22:49:33.198 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T22:49:33.198 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T22:49:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-03-11T22:49:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-11T22:49:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T22:49:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T22:49:33.270 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:49:33.270 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 8 2024-03-11T22:49:41.271 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T22:49:41.272 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T22:49:41.274 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:49:41.274 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-11T22:49:41.274 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T22:49:41.274 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:49:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:49:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T22:49:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T22:49:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:49:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:49:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:49:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T22:49:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T22:49:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:49:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T22:49:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:49:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T22:49:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T22:49:41.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:49:41.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:49:41.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T22:49:41.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T22:49:41.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:49:41.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:49:41.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:41.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:49:41.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T22:49:41.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:49:41.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:49:41.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T22:49:41.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T22:49:41.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T22:49:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:49:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:49:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:49:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-11T22:49:41.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-11T22:49:41.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T22:49:41.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T22:49:41.377 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:49:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-03-11T22:49:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T22:49:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:49:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:49:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T22:49:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T22:49:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:49:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:49:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:49:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T22:49:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T22:49:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:49:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T22:49:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T22:49:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T22:49:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T22:49:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:49:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:49:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T22:49:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T22:49:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:49:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:49:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T22:49:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T22:49:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:49:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:49:41.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T22:49:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T22:49:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T22:49:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T22:49:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T22:49:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T22:49:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-03-11T22:49:41.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-11T22:49:41.470 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T22:49:41.470 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T22:49:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:49:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-03-11T22:49:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T22:49:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:49:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:49:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T22:49:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T22:49:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:49:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:49:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:49:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T22:49:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T22:49:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:49:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T22:49:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T22:49:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T22:49:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T22:49:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:49:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:49:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T22:49:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T22:49:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:49:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:49:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T22:49:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T22:49:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:49:41.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:49:41.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T22:49:41.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T22:49:41.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T22:49:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T22:49:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T22:49:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T22:49:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-03-11T22:49:41.563 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-11T22:49:41.563 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T22:49:41.563 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T22:49:41.566 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:49:41.567 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-03-11T22:49:41.567 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:3 2024-03-11T22:49:41.567 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:49:41.567 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:49:41.567 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-03-11T22:49:41.567 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:3 2024-03-11T22:49:41.567 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:49:41.567 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:49:41.567 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:49:41.567 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=3 2024-03-11T22:49:41.567 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 3 = cluster1:3 ']' 2024-03-11T22:49:41.567 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:49:41.567 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=3 2024-03-11T22:49:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T22:49:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-03-11T22:49:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:3 2024-03-11T22:49:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:49:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:49:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-03-11T22:49:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:3 2024-03-11T22:49:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:49:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:49:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T22:49:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 3 = cluster1:3 ']' 2024-03-11T22:49:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:49:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:49:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=3 2024-03-11T22:49:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=3 2024-03-11T22:49:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-03-11T22:49:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T22:49:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T22:49:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T22:49:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-03-11T22:49:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-11T22:49:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T22:49:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T22:49:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:49:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 8 2024-03-11T22:49:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T22:49:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T22:49:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:49:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-11T22:49:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T22:49:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:49:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:49:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T22:49:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T22:49:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:49:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:49:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:49:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T22:49:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T22:49:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:49:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T22:49:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:49:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T22:49:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T22:49:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:49:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:49:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T22:49:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T22:49:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:49:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:49:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:49:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:49:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T22:49:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:49:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:49:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T22:49:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T22:49:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T22:49:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:49:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:49:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:49:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-11T22:49:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T22:49:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T22:49:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T22:49:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T22:49:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T22:49:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T22:49:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T22:49:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:49:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ clean_snap_name= 2024-03-11T22:49:49.780 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ seq 1 10 2024-03-11T22:49:49.780 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T22:49:49.780 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror test 2024-03-11T22:49:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:49:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:49:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:49:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T22:49:49.782 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=12 2024-03-11T22:49:49.782 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T22:49:49.782 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-03-11T22:49:49.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 16221184 bytes 2024-03-11T22:49:50.086 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 68678656 bytes 2024-03-11T22:49:50.147 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 69291008 bytes 2024-03-11T22:49:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 85036032 bytes 2024-03-11T22:49:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 149471232 bytes 2024-03-11T22:49:50.514 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 168261632 bytes 2024-03-11T22:49:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 207460352 bytes 2024-03-11T22:49:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 233653248 bytes 2024-03-11T22:49:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 254723072 bytes 2024-03-11T22:49:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 311488512 bytes 2024-03-11T22:49:51.219 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 335731712 bytes 2024-03-11T22:49:51.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 362115072 bytes 2024-03-11T22:49:51.515 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 421071872 bytes 2024-03-11T22:49:51.724 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 493010944 bytes 2024-03-11T22:49:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 565810176 bytes 2024-03-11T22:49:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 616386560 bytes 2024-03-11T22:49:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 642704384 bytes 2024-03-11T22:49:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 686215168 bytes 2024-03-11T22:49:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 740005888 bytes 2024-03-11T22:49:52.613 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 762672128 bytes 2024-03-11T22:49:52.768 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 806535168 bytes 2024-03-11T22:49:52.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 848125952 bytes 2024-03-11T22:49:53.092 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 906587136 bytes 2024-03-11T22:49:53.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 965055488 bytes 2024-03-11T22:49:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 987468800 bytes 2024-03-11T22:49:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1002718208 bytes 2024-03-11T22:49:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1009800192 bytes 2024-03-11T22:49:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1053818880 bytes 2024-03-11T22:49:53.865 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1108646912 bytes 2024-03-11T22:49:54.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1156335616 bytes 2024-03-11T22:49:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1208180736 bytes 2024-03-11T22:49:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1265513472 bytes 2024-03-11T22:49:54.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1376610304 bytes 2024-03-11T22:49:54.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1382300672 bytes 2024-03-11T22:49:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1446802432 bytes 2024-03-11T22:49:54.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1450931200 bytes 2024-03-11T22:49:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1510878208 bytes 2024-03-11T22:49:55.269 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1530328064 bytes 2024-03-11T22:49:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1593796608 bytes 2024-03-11T22:49:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1637013504 bytes 2024-03-11T22:49:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1652847616 bytes 2024-03-11T22:49:55.884 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1700826112 bytes 2024-03-11T22:49:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1731074048 bytes 2024-03-11T22:49:56.178 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 1774636032 bytes 2024-03-11T22:49:56.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 1809126400 bytes 2024-03-11T22:49:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 1816365056 bytes 2024-03-11T22:49:56.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 1878543360 bytes 2024-03-11T22:49:56.789 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 1973477376 bytes 2024-03-11T22:49:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 2016195584 bytes 2024-03-11T22:49:57.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 2027758592 bytes 2024-03-11T22:49:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5100 IOs, 2030899200 bytes 2024-03-11T22:49:57.303 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5200 IOs, 2086980608 bytes 2024-03-11T22:49:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5300 IOs, 2092701696 bytes 2024-03-11T22:49:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5400 IOs, 2132361216 bytes 2024-03-11T22:49:57.649 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8979451160888803) > weight (0.25) 2024-03-11T22:49:57.649 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.933428096778959) > weight (0.25) 2024-03-11T22:49:57.649 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.27681076650096603) > weight (0.25) 2024-03-11T22:49:57.649 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.8564442640085594) > weight (0.25) 2024-03-11T22:49:57.649 INFO:tasks.rbd_mirror_thrash:waiting for 65 secs before thrashing 2024-03-11T22:49:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5500 IOs, 2192388096 bytes 2024-03-11T22:49:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5600 IOs, 2199175168 bytes 2024-03-11T22:49:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5700 IOs, 2219620352 bytes 2024-03-11T22:49:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5800 IOs, 2248820736 bytes 2024-03-11T22:49:58.173 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5900 IOs, 2291154944 bytes 2024-03-11T22:49:58.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6000 IOs, 2340165632 bytes 2024-03-11T22:49:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6100 IOs, 2431130624 bytes 2024-03-11T22:49:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6200 IOs, 2456299520 bytes 2024-03-11T22:49:58.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6300 IOs, 2477668352 bytes 2024-03-11T22:49:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6400 IOs, 2558771200 bytes 2024-03-11T22:49:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6500 IOs, 2588999680 bytes 2024-03-11T22:49:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6600 IOs, 2630768640 bytes 2024-03-11T22:49:59.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6700 IOs, 2631384064 bytes 2024-03-11T22:49:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6800 IOs, 2680904704 bytes 2024-03-11T22:49:59.669 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6900 IOs, 2698678272 bytes 2024-03-11T22:49:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7000 IOs, 2746089472 bytes 2024-03-11T22:49:59.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7100 IOs, 2760979456 bytes 2024-03-11T22:50:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7200 IOs, 2780243968 bytes 2024-03-11T22:50:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7300 IOs, 2814478336 bytes 2024-03-11T22:50:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7400 IOs, 2815102976 bytes 2024-03-11T22:50:00.400 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7500 IOs, 2851083264 bytes 2024-03-11T22:50:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7600 IOs, 2851712000 bytes 2024-03-11T22:50:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7700 IOs, 2891603968 bytes 2024-03-11T22:50:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7800 IOs, 2943986688 bytes 2024-03-11T22:50:00.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7900 IOs, 2992992256 bytes 2024-03-11T22:50:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:8000 IOs, 3064250368 bytes 2024-03-11T22:50:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:8100 IOs, 3151248384 bytes 2024-03-11T22:50:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:8200 IOs, 3194592256 bytes 2024-03-11T22:50:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:8300 IOs, 3234808832 bytes 2024-03-11T22:50:01.787 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T22:50:01.787 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T22:50:01.787 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T22:50:01.787 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:50:01.787 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-11T22:50:01.787 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:50:01.787 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:50:01.787 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:50:01.787 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:50:01.787 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local description_pattern= 2024-03-11T22:50:01.787 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local service_pattern= 2024-03-11T22:50:01.788 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:50:01.788 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T22:50:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-11T22:50:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:50:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:50:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:50:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:50:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local description_pattern= 2024-03-11T22:50:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local service_pattern= 2024-03-11T22:50:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status 2024-03-11T22:50:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-11T22:50:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-11T22:50:02.839 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status='test: 2024-03-11T22:50:02.839 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: global_id: d6055756-c87a-420f-9f23-02c2bbaa6857 2024-03-11T22:50:02.839 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: state: down+unknown 2024-03-11T22:50:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: description: status not found 2024-03-11T22:50:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: last_update: 2024-03-11T22:50:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: peer_sites: 2024-03-11T22:50:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: name: cluster2 2024-03-11T22:50:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: state: down+unknown 2024-03-11T22:50:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: description: status not found 2024-03-11T22:50:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: last_update: ' 2024-03-11T22:50:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^ state: .*up+replaying' 2024-03-11T22:50:02.841 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 1 2024-03-11T22:50:02.841 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:50:02.841 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 2 2024-03-11T22:50:04.842 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-11T22:50:04.842 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:50:04.842 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:50:04.842 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:50:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:50:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local description_pattern= 2024-03-11T22:50:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local service_pattern= 2024-03-11T22:50:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status 2024-03-11T22:50:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-11T22:50:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-11T22:50:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status='test: 2024-03-11T22:50:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: global_id: d6055756-c87a-420f-9f23-02c2bbaa6857 2024-03-11T22:50:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: state: down+unknown 2024-03-11T22:50:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: description: status not found 2024-03-11T22:50:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: last_update: 2024-03-11T22:50:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: peer_sites: 2024-03-11T22:50:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: name: cluster2 2024-03-11T22:50:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: state: down+unknown 2024-03-11T22:50:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: description: status not found 2024-03-11T22:50:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: last_update: ' 2024-03-11T22:50:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^ state: .*up+replaying' 2024-03-11T22:50:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 1 2024-03-11T22:50:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:50:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 4 2024-03-11T22:50:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-11T22:50:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:50:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:50:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:50:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:50:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local description_pattern= 2024-03-11T22:50:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local service_pattern= 2024-03-11T22:50:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status 2024-03-11T22:50:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-11T22:50:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-11T22:50:08.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status='test: 2024-03-11T22:50:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: global_id: d6055756-c87a-420f-9f23-02c2bbaa6857 2024-03-11T22:50:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: state: down+unknown 2024-03-11T22:50:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: description: status not found 2024-03-11T22:50:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: last_update: 2024-03-11T22:50:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: peer_sites: 2024-03-11T22:50:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: name: cluster2 2024-03-11T22:50:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: state: down+unknown 2024-03-11T22:50:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: description: status not found 2024-03-11T22:50:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: last_update: ' 2024-03-11T22:50:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^ state: .*up+replaying' 2024-03-11T22:50:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 1 2024-03-11T22:50:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:50:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 8 2024-03-11T22:50:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-11T22:50:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:50:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:50:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:50:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:50:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local description_pattern= 2024-03-11T22:50:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local service_pattern= 2024-03-11T22:50:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status 2024-03-11T22:50:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-11T22:50:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-11T22:50:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status='test: 2024-03-11T22:50:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: global_id: d6055756-c87a-420f-9f23-02c2bbaa6857 2024-03-11T22:50:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: state: up+replaying 2024-03-11T22:50:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1710197363,"remote_snapshot_timestamp":1710197363,"replay_state":"idle"} 2024-03-11T22:50:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: service: mirror.0 on smithi132 2024-03-11T22:50:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: last_update: 2024-03-11 22:50:09 2024-03-11T22:50:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: peer_sites: 2024-03-11T22:50:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: name: cluster2 2024-03-11T22:50:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: state: up+stopped 2024-03-11T22:50:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: description: local image is primary 2024-03-11T22:50:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: last_update: 2024-03-11 22:50:09' 2024-03-11T22:50:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^ state: .*up+replaying' 2024-03-11T22:50:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: state: up+replaying 2024-03-11T22:50:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^ description: .*' 2024-03-11T22:50:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1710197363,"remote_snapshot_timestamp":1710197363,"replay_state":"idle"} 2024-03-11T22:50:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T22:50:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo up+replaying 2024-03-11T22:50:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^up+' 2024-03-11T22:50:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:up+replaying 2024-03-11T22:50:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep 'service: *mirror..* on ' 2024-03-11T22:50:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: service: mirror.0 on smithi132 2024-03-11T22:50:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local last_update 2024-03-11T22:50:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-11T22:50:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ last_update='2024-03-11 22:50:09' 2024-03-11T22:50:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-11 22:50:09' 2024-03-11T22:50:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:50:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:50:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:50:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:50:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local 'prev_last_update=2024-03-11 22:50:09' 2024-03-11T22:50:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status 2024-03-11T22:50:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-11T22:50:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-11T22:50:17.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status='OKOKOK142104270mirror.1smithi13219.0.0-1979-gb117a47afalseOK42114267mirror.0smithi13219.0.0-1979-gb117a47atrueOK42144271mirror.2smithi13219.0.0-1979-gb117a47afalseOK42174275mirror.3smithi13219.0.0-1979-gb117a47afalseOKtestd6055756-c87a-420f-9f23-02c2bbaa6857up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1710197363,"remote_snapshot_timestamp":1710197363,"replay_state":"idle"}42114267mirror.0smithi1322024-03-11 22:50:09cluster2baf1db9c-11db-489a-a01f-16a50bade1d5up+stoppedlocal image is primary2024-03-11 22:50:09' 2024-03-11T22:50:17.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local last_update state 2024-03-11T22:50:17.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-11T22:50:17.113 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ last_update='2024-03-11 22:50:09' 2024-03-11T22:50:17.114 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-11T22:50:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ state=up+replaying 2024-03-11T22:50:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo up+replaying 2024-03-11T22:50:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep up+replaying 2024-03-11T22:50:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:up+replaying 2024-03-11T22:50:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:50:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:50:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap1 2024-03-11T22:50:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror test snap1 2024-03-11T22:50:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:50:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:50:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:50:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap1 2024-03-11T22:50:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2024-03-11T22:50:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-11T22:50:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:50:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:50:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:50:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-11T22:50:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:50:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:50:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:50:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T22:50:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T22:50:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:50:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T22:50:31.695 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-11T22:50:31.695 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:50:31.695 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:50:31.695 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:50:31.695 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T22:50:31.695 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T22:50:31.695 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T22:50:31.695 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-11T22:50:31.696 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T22:50:31.696 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T22:50:31.696 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:50:31.696 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T22:50:31.696 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T22:50:31.696 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T22:50:31.696 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:50:31.696 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T22:50:31.699 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T22:50:31.699 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T22:50:31.699 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:50:31.699 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-11T22:50:31.699 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T22:50:31.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:50:31.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:50:31.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T22:50:31.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T22:50:31.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:50:31.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:50:31.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:50:31.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T22:50:31.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T22:50:31.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:50:31.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:50:31.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T22:50:31.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:50:31.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T22:50:31.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T22:50:31.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:50:31.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:50:31.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T22:50:31.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T22:50:31.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:50:31.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:50:31.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:50:31.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:50:31.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T22:50:31.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:50:31.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:50:31.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T22:50:31.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T22:50:31.702 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T22:50:31.719 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:50:31.719 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:50:31.719 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:50:31.719 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-11T22:50:31.807 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T22:50:31.807 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T22:50:31.808 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T22:50:31.808 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T22:50:31.808 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T22:50:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T22:50:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T22:50:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:50:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-11T22:50:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T22:50:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:50:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:50:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:50:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T22:50:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T22:50:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-03-11T22:50:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T22:50:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:50:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:50:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:50:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-test.status 2024-03-11T22:50:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-test.status 2024-03-11T22:50:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T22:50:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.status 2024-03-11T22:50:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-test.status 2024-03-11T22:50:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-test.status 2024-03-11T22:50:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T22:50:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:50:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-03-11T22:50:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:50:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:50:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:50:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-03-11T22:50:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 5 2024-03-11T22:50:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.status 2024-03-11T22:50:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:50:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:50:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:50:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.status 2024-03-11T22:50:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-03-11T22:50:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:50:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T22:50:32.754 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=5 2024-03-11T22:50:32.754 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:50:32.754 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:50:32.754 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T22:50:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:50:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:50:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:50:32.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:50:32.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:50:32.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:50:32.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:50:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:50:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:50:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T22:50:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:50:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T22:50:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T22:50:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 5 = '' 2024-03-11T22:50:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:50:33.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T22:50:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:50:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:50:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:50:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:50:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:50:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:50:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:50:33.456 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:50:33.456 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:50:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T22:50:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:50:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T22:50:33.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T22:50:33.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 5 = '' 2024-03-11T22:50:33.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:50:33.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T22:50:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:50:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:50:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:50:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:50:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:50:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:50:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:50:34.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:50:34.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:50:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T22:50:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:50:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T22:50:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T22:50:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 5 = '' 2024-03-11T22:50:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:50:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T22:50:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:50:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:50:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:50:35.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:50:35.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:50:35.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:50:35.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:50:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:50:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=5 2024-03-11T22:50:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 5 = 5 2024-03-11T22:50:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:50:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2024-03-11T22:50:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:50:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:50:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:50:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap1 2024-03-11T22:50:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T22:50:36.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T22:50:36.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T22:50:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2024-03-11T22:50:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'test': 2024-03-11T22:50:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 512 MiB in 128 objects 2024-03-11T22:50:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T22:50:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T22:50:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 10ab50368aa4 2024-03-11T22:50:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.10ab50368aa4 2024-03-11T22:50:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T22:50:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T22:50:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T22:50:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T22:50:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 22:49:42 2024 2024-03-11T22:50:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 22:49:42 2024 2024-03-11T22:50:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 22:49:42 2024 2024-03-11T22:50:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T22:50:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T22:50:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T22:50:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: d6055756-c87a-420f-9f23-02c2bbaa6857 2024-03-11T22:50:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T22:50:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:50:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T22:50:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap1 2024-03-11T22:50:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:50:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:50:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap1 2024-03-11T22:50:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:50:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:50:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:50:37.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:50:37.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:50:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:50:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:50:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:50:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:50:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:50:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:50:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ clean_snap_name=snap1-clean 2024-03-11T22:50:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror test snap1-clean 2024-03-11T22:50:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:50:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:50:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:50:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap1-clean 2024-03-11T22:50:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2024-03-11T22:50:39.707 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T22:50:39.707 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror test 2024-03-11T22:50:39.707 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:50:39.707 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:50:39.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:50:39.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T22:50:39.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=12 2024-03-11T22:50:39.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T22:50:39.710 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-03-11T22:50:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 14552064 bytes 2024-03-11T22:50:39.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 29366272 bytes 2024-03-11T22:50:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 52757504 bytes 2024-03-11T22:50:40.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 120122368 bytes 2024-03-11T22:50:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 144476160 bytes 2024-03-11T22:50:40.543 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 169516032 bytes 2024-03-11T22:50:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 207230976 bytes 2024-03-11T22:50:40.760 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 207923200 bytes 2024-03-11T22:50:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 233590784 bytes 2024-03-11T22:50:41.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 269872128 bytes 2024-03-11T22:50:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 312474624 bytes 2024-03-11T22:50:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 318902272 bytes 2024-03-11T22:50:41.412 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 365270016 bytes 2024-03-11T22:50:41.523 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 416856064 bytes 2024-03-11T22:50:41.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 440620032 bytes 2024-03-11T22:50:41.814 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 470198272 bytes 2024-03-11T22:50:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 511023104 bytes 2024-03-11T22:50:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 556866560 bytes 2024-03-11T22:50:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 557473792 bytes 2024-03-11T22:50:42.399 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 630341632 bytes 2024-03-11T22:50:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 637775872 bytes 2024-03-11T22:50:42.555 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 659437568 bytes 2024-03-11T22:50:42.727 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 705694720 bytes 2024-03-11T22:50:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 738760704 bytes 2024-03-11T22:50:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 793508864 bytes 2024-03-11T22:50:43.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 853066752 bytes 2024-03-11T22:50:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 907324416 bytes 2024-03-11T22:50:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 978324480 bytes 2024-03-11T22:50:43.686 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 988804096 bytes 2024-03-11T22:50:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1048306688 bytes 2024-03-11T22:50:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1088079872 bytes 2024-03-11T22:50:44.134 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1112714240 bytes 2024-03-11T22:50:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1196706816 bytes 2024-03-11T22:50:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1229858816 bytes 2024-03-11T22:50:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1269931008 bytes 2024-03-11T22:50:44.787 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1302157312 bytes 2024-03-11T22:50:44.871 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1313164288 bytes 2024-03-11T22:50:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1401695232 bytes 2024-03-11T22:50:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1411689472 bytes 2024-03-11T22:50:45.299 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1433957376 bytes 2024-03-11T22:50:45.515 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1505630208 bytes 2024-03-11T22:50:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1556800512 bytes 2024-03-11T22:50:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1574228992 bytes 2024-03-11T22:50:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 1591104512 bytes 2024-03-11T22:50:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 1667346432 bytes 2024-03-11T22:50:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 1667931136 bytes 2024-03-11T22:50:46.409 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 1736875008 bytes 2024-03-11T22:50:46.545 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 1772154880 bytes 2024-03-11T22:50:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 1772747776 bytes 2024-03-11T22:50:46.799 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 1853296640 bytes 2024-03-11T22:50:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5100 IOs, 1897134080 bytes 2024-03-11T22:50:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5200 IOs, 1968328704 bytes 2024-03-11T22:50:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5300 IOs, 2048898048 bytes 2024-03-11T22:50:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5400 IOs, 2104474624 bytes 2024-03-11T22:50:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5500 IOs, 2151909376 bytes 2024-03-11T22:50:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5600 IOs, 2174873600 bytes 2024-03-11T22:50:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5700 IOs, 2244947968 bytes 2024-03-11T22:50:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5800 IOs, 2297521152 bytes 2024-03-11T22:50:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5900 IOs, 2324147200 bytes 2024-03-11T22:50:48.578 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6000 IOs, 2398571520 bytes 2024-03-11T22:50:48.650 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8686788398075632) > weight (0.25) 2024-03-11T22:50:48.650 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.25436214630709586) > weight (0.25) 2024-03-11T22:50:48.650 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.4623012381714606) > weight (0.25) 2024-03-11T22:50:48.651 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6833485931759454) > weight (0.25) 2024-03-11T22:50:48.651 INFO:tasks.rbd_mirror_thrash:waiting for 107 secs before thrashing 2024-03-11T22:50:48.804 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6100 IOs, 2460539904 bytes 2024-03-11T22:50:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6200 IOs, 2467523584 bytes 2024-03-11T22:50:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6300 IOs, 2511001600 bytes 2024-03-11T22:50:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6400 IOs, 2573271040 bytes 2024-03-11T22:50:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6500 IOs, 2631100416 bytes 2024-03-11T22:50:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6600 IOs, 2651596800 bytes 2024-03-11T22:50:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6700 IOs, 2734331904 bytes 2024-03-11T22:50:49.829 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6800 IOs, 2752680960 bytes 2024-03-11T22:50:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6900 IOs, 2821863424 bytes 2024-03-11T22:50:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7000 IOs, 2877065216 bytes 2024-03-11T22:50:50.453 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7100 IOs, 2970267648 bytes 2024-03-11T22:50:50.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7200 IOs, 2998922240 bytes 2024-03-11T22:50:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7300 IOs, 3034877952 bytes 2024-03-11T22:50:50.932 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7400 IOs, 3103807488 bytes 2024-03-11T22:50:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7500 IOs, 3172509696 bytes 2024-03-11T22:50:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7600 IOs, 3220828160 bytes 2024-03-11T22:50:51.430 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7700 IOs, 3249396736 bytes 2024-03-11T22:50:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7800 IOs, 3302721536 bytes 2024-03-11T22:50:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T22:50:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T22:50:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T22:50:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:50:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-11T22:50:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:50:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:50:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:50:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:50:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local description_pattern= 2024-03-11T22:50:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local service_pattern= 2024-03-11T22:50:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:50:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T22:50:52.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-11T22:50:52.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:50:52.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:50:52.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:50:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:50:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local description_pattern= 2024-03-11T22:50:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local service_pattern= 2024-03-11T22:50:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status 2024-03-11T22:50:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-11T22:50:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-11T22:50:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status='test: 2024-03-11T22:50:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: global_id: d6055756-c87a-420f-9f23-02c2bbaa6857 2024-03-11T22:50:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: state: up+replaying 2024-03-11T22:50:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: description: replaying, {"bytes_per_second":17892454.4,"bytes_per_snapshot":268386816.0,"last_snapshot_bytes":536773632,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1710197432,"remote_snapshot_timestamp":1710197432,"replay_state":"idle"} 2024-03-11T22:50:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: service: mirror.0 on smithi132 2024-03-11T22:50:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: last_update: 2024-03-11 22:50:39 2024-03-11T22:50:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: peer_sites: 2024-03-11T22:50:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: name: cluster2 2024-03-11T22:50:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: state: up+stopped 2024-03-11T22:50:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: description: local image is primary 2024-03-11T22:50:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: last_update: 2024-03-11 22:50:39' 2024-03-11T22:50:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^ state: .*up+replaying' 2024-03-11T22:50:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: state: up+replaying 2024-03-11T22:50:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^ description: .*' 2024-03-11T22:50:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: description: replaying, {"bytes_per_second":17892454.4,"bytes_per_snapshot":268386816.0,"last_snapshot_bytes":536773632,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1710197432,"remote_snapshot_timestamp":1710197432,"replay_state":"idle"} 2024-03-11T22:50:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T22:50:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo up+replaying 2024-03-11T22:50:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^up+' 2024-03-11T22:50:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:up+replaying 2024-03-11T22:50:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep 'service: *mirror..* on ' 2024-03-11T22:50:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: service: mirror.0 on smithi132 2024-03-11T22:50:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local last_update 2024-03-11T22:50:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-11T22:50:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ last_update='2024-03-11 22:50:39' 2024-03-11T22:50:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-11 22:50:39' 2024-03-11T22:50:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:50:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:50:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:50:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:50:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local 'prev_last_update=2024-03-11 22:50:39' 2024-03-11T22:50:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status 2024-03-11T22:50:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-11T22:50:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-11T22:50:52.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status='OKOKOK142104270mirror.1smithi13219.0.0-1979-gb117a47afalseOK42114267mirror.0smithi13219.0.0-1979-gb117a47atrueOK42144271mirror.2smithi13219.0.0-1979-gb117a47afalseOK42174275mirror.3smithi13219.0.0-1979-gb117a47afalseOKtestd6055756-c87a-420f-9f23-02c2bbaa6857up+replayingreplaying, {"bytes_per_second":17892454.4,"bytes_per_snapshot":268386816.0,"last_snapshot_bytes":536773632,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1710197432,"remote_snapshot_timestamp":1710197432,"replay_state":"idle"}42114267mirror.0smithi1322024-03-11 22:50:39cluster2baf1db9c-11db-489a-a01f-16a50bade1d5up+stoppedlocal image is primary2024-03-11 22:50:39' 2024-03-11T22:50:52.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local last_update state 2024-03-11T22:50:52.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-11T22:50:52.842 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ last_update='2024-03-11 22:50:39' 2024-03-11T22:50:52.842 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-11T22:50:52.844 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ state=up+replaying 2024-03-11T22:50:52.844 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo up+replaying 2024-03-11T22:50:52.844 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep up+replaying 2024-03-11T22:50:52.845 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:up+replaying 2024-03-11T22:50:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:50:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:50:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap2 2024-03-11T22:50:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror test snap2 2024-03-11T22:50:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:50:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:50:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:50:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap2 2024-03-11T22:50:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2024-03-11T22:51:02.650 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3301044322083555) > weight (0.25) 2024-03-11T22:51:02.651 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7281809556696955) > weight (0.25) 2024-03-11T22:51:02.651 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-03-11T22:51:02.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-03-11T22:51:02.652 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-03-11T22:51:02.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi132.stderr:2024-03-11T22:51:02.648+0000 7fb8eed10640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 40009) UID: 1000 2024-03-11T22:51:09.653 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-03-11T22:51:09.653 INFO:teuthology.orchestra.run:waiting for 600 2024-03-11T22:51:09.654 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-03-11T22:51:09.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-03-11T22:51:09.654 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-03-11T22:51:09.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-03-11T22:51:09.657 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2024-03-11T22:51:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-11T22:51:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:51:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:51:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:51:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-11T22:51:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:51:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:51:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:51:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T22:51:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T22:51:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:51:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T22:51:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-11T22:51:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:51:22.814 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:51:22.814 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:51:22.814 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T22:51:22.814 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T22:51:22.814 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T22:51:22.814 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-11T22:51:22.814 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T22:51:22.814 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:51:22.814 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T22:51:22.814 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T22:51:22.814 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T22:51:22.814 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T22:51:22.814 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:51:22.815 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T22:51:22.815 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T22:51:22.815 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T22:51:22.816 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:51:22.816 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-11T22:51:22.816 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T22:51:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:51:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:51:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T22:51:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T22:51:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:51:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:51:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:51:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T22:51:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T22:51:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:51:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:51:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T22:51:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:51:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T22:51:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T22:51:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:51:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:51:22.819 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T22:51:22.819 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T22:51:22.819 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:51:22.819 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:51:22.819 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:51:22.819 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:51:22.819 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T22:51:22.819 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:51:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:51:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T22:51:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T22:51:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T22:51:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:51:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:51:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:51:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-11T22:51:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T22:51:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T22:51:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T22:51:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T22:51:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T22:51:22.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T22:51:22.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T22:51:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:51:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-11T22:51:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T22:51:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:51:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:51:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:51:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T22:51:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T22:51:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-03-11T22:51:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T22:51:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:51:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:51:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:51:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-test.status 2024-03-11T22:51:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-test.status 2024-03-11T22:51:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T22:51:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.status 2024-03-11T22:51:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-test.status 2024-03-11T22:51:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-test.status 2024-03-11T22:51:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T22:51:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:51:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-03-11T22:51:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:51:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:51:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:51:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-03-11T22:51:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 8 2024-03-11T22:51:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.status 2024-03-11T22:51:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:51:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:51:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:51:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.status 2024-03-11T22:51:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-03-11T22:51:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:51:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T22:51:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=8 2024-03-11T22:51:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:51:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:51:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T22:51:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:51:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:51:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:51:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:51:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:51:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:51:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:51:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:51:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:51:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T22:51:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:51:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T22:51:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T22:51:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 8 = '' 2024-03-11T22:51:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:51:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T22:51:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:51:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:51:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:51:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:51:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:51:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:51:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:51:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:51:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:51:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T22:51:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:51:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T22:51:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T22:51:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 8 = '' 2024-03-11T22:51:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:51:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T22:51:26.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:51:26.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:51:26.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:51:26.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:51:26.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:51:26.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:51:26.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:51:26.430 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:51:26.430 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:51:26.432 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T22:51:26.432 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:51:26.432 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T22:51:26.432 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T22:51:26.432 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 8 = '' 2024-03-11T22:51:26.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:51:26.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T22:51:28.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:51:28.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:51:28.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:51:28.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:51:28.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:51:28.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:51:28.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:51:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:51:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=8 2024-03-11T22:51:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 8 = 8 2024-03-11T22:51:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:51:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2024-03-11T22:51:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:51:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:51:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:51:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap2 2024-03-11T22:51:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T22:51:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T22:51:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T22:51:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2024-03-11T22:51:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'test': 2024-03-11T22:51:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 512 MiB in 128 objects 2024-03-11T22:51:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T22:51:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 4 2024-03-11T22:51:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 10ab50368aa4 2024-03-11T22:51:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.10ab50368aa4 2024-03-11T22:51:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T22:51:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T22:51:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T22:51:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T22:51:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 22:49:42 2024 2024-03-11T22:51:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 22:49:42 2024 2024-03-11T22:51:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 22:49:42 2024 2024-03-11T22:51:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T22:51:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T22:51:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T22:51:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: d6055756-c87a-420f-9f23-02c2bbaa6857 2024-03-11T22:51:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T22:51:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:51:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snap1-clean ']' 2024-03-11T22:51:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap1-clean 2024-03-11T22:51:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:51:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:51:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap1-clean 2024-03-11T22:51:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:51:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:51:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:51:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:51:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:51:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:51:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:51:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:51:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:51:30.727 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:51:30.844 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:51:30.844 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap2 2024-03-11T22:51:30.844 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:51:30.844 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:51:30.844 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap2 2024-03-11T22:51:30.845 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:51:30.845 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:51:30.845 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:51:30.845 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:51:30.845 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:51:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:51:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:51:32.254 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:51:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:51:32.472 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:51:32.587 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:51:32.587 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ clean_snap_name=snap2-clean 2024-03-11T22:51:32.587 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror test snap2-clean 2024-03-11T22:51:32.587 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:51:32.587 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:51:32.587 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:51:32.587 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap2-clean 2024-03-11T22:51:32.588 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2024-03-11T22:51:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T22:51:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror test 2024-03-11T22:51:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:51:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:51:33.826 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:51:33.826 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T22:51:33.827 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=10 2024-03-11T22:51:33.827 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T22:51:33.827 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-03-11T22:51:34.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 42985472 bytes 2024-03-11T22:51:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 110536704 bytes 2024-03-11T22:51:34.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 154573824 bytes 2024-03-11T22:51:34.513 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 155174912 bytes 2024-03-11T22:51:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 193168384 bytes 2024-03-11T22:51:34.789 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 227123200 bytes 2024-03-11T22:51:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 255781888 bytes 2024-03-11T22:51:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 323759104 bytes 2024-03-11T22:51:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 366920704 bytes 2024-03-11T22:51:35.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 411229184 bytes 2024-03-11T22:51:35.566 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 478808064 bytes 2024-03-11T22:51:35.821 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 552132608 bytes 2024-03-11T22:51:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 591485952 bytes 2024-03-11T22:51:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 685434880 bytes 2024-03-11T22:51:36.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 701843456 bytes 2024-03-11T22:51:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 722746368 bytes 2024-03-11T22:51:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 761281536 bytes 2024-03-11T22:51:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 785998848 bytes 2024-03-11T22:51:36.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 865052672 bytes 2024-03-11T22:51:37.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 882706432 bytes 2024-03-11T22:51:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 940624896 bytes 2024-03-11T22:51:37.331 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 959257600 bytes 2024-03-11T22:51:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 959858688 bytes 2024-03-11T22:51:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 981246976 bytes 2024-03-11T22:51:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1029246976 bytes 2024-03-11T22:51:37.928 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1102371840 bytes 2024-03-11T22:51:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1147245568 bytes 2024-03-11T22:51:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1175745536 bytes 2024-03-11T22:51:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1214040064 bytes 2024-03-11T22:51:38.530 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1263489024 bytes 2024-03-11T22:51:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1325807616 bytes 2024-03-11T22:51:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1383586816 bytes 2024-03-11T22:51:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1389529088 bytes 2024-03-11T22:51:39.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1419496448 bytes 2024-03-11T22:51:39.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1533885440 bytes 2024-03-11T22:51:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1625027584 bytes 2024-03-11T22:51:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1720979456 bytes 2024-03-11T22:51:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1725351936 bytes 2024-03-11T22:51:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1770394624 bytes 2024-03-11T22:51:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1783532544 bytes 2024-03-11T22:51:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1886625792 bytes 2024-03-11T22:51:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1963828224 bytes 2024-03-11T22:51:40.908 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1995064320 bytes 2024-03-11T22:51:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 1995649024 bytes 2024-03-11T22:51:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 2052867072 bytes 2024-03-11T22:51:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 2157544448 bytes 2024-03-11T22:51:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 2222209024 bytes 2024-03-11T22:51:41.798 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 2276869120 bytes 2024-03-11T22:51:41.900 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 2297078784 bytes 2024-03-11T22:51:41.980 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 2315142144 bytes 2024-03-11T22:51:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5100 IOs, 2332219392 bytes 2024-03-11T22:51:42.207 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5200 IOs, 2349291520 bytes 2024-03-11T22:51:42.300 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5300 IOs, 2360805376 bytes 2024-03-11T22:51:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5400 IOs, 2384367616 bytes 2024-03-11T22:51:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5500 IOs, 2418642944 bytes 2024-03-11T22:51:42.778 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5600 IOs, 2521015296 bytes 2024-03-11T22:51:43.014 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5700 IOs, 2574909440 bytes 2024-03-11T22:51:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5800 IOs, 2616577024 bytes 2024-03-11T22:51:43.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5900 IOs, 2673213440 bytes 2024-03-11T22:51:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6000 IOs, 2696126464 bytes 2024-03-11T22:51:43.561 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6100 IOs, 2708610048 bytes 2024-03-11T22:51:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6200 IOs, 2718343168 bytes 2024-03-11T22:51:43.775 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6300 IOs, 2749514752 bytes 2024-03-11T22:51:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T22:51:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T22:51:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T22:51:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:51:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-11T22:51:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:51:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:51:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:51:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:51:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local description_pattern= 2024-03-11T22:51:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local service_pattern= 2024-03-11T22:51:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:51:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T22:51:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-11T22:51:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:51:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:51:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:51:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:51:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local description_pattern= 2024-03-11T22:51:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local service_pattern= 2024-03-11T22:51:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status 2024-03-11T22:51:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-11T22:51:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-11T22:51:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status='test: 2024-03-11T22:51:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: global_id: d6055756-c87a-420f-9f23-02c2bbaa6857 2024-03-11T22:51:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: state: up+replaying 2024-03-11T22:51:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: description: replaying, {"bytes_per_second":17685504.0,"bytes_per_snapshot":533669376.0,"last_snapshot_bytes":530565120,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1710197483,"remote_snapshot_timestamp":1710197483,"replay_state":"idle"} 2024-03-11T22:51:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: service: mirror.0 on smithi132 2024-03-11T22:51:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: last_update: 2024-03-11 22:51:39 2024-03-11T22:51:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: peer_sites: 2024-03-11T22:51:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: name: cluster2 2024-03-11T22:51:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: state: up+stopped 2024-03-11T22:51:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: description: local image is primary 2024-03-11T22:51:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: last_update: 2024-03-11 22:51:39' 2024-03-11T22:51:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^ state: .*up+replaying' 2024-03-11T22:51:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: state: up+replaying 2024-03-11T22:51:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^ description: .*' 2024-03-11T22:51:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: description: replaying, {"bytes_per_second":17685504.0,"bytes_per_snapshot":533669376.0,"last_snapshot_bytes":530565120,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1710197483,"remote_snapshot_timestamp":1710197483,"replay_state":"idle"} 2024-03-11T22:51:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T22:51:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo up+replaying 2024-03-11T22:51:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^up+' 2024-03-11T22:51:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:up+replaying 2024-03-11T22:51:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep 'service: *mirror..* on ' 2024-03-11T22:51:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: service: mirror.0 on smithi132 2024-03-11T22:51:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local last_update 2024-03-11T22:51:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-11T22:51:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ last_update='2024-03-11 22:51:39' 2024-03-11T22:51:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-11 22:51:39' 2024-03-11T22:51:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:51:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:51:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:51:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:51:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local 'prev_last_update=2024-03-11 22:51:39' 2024-03-11T22:51:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status 2024-03-11T22:51:44.895 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-11T22:51:44.895 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-11T22:51:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status='OKOKOK142104270mirror.1smithi13219.0.0-1979-gb117a47afalseOK42114267mirror.0smithi13219.0.0-1979-gb117a47atrueOK42174275mirror.3smithi13219.0.0-1979-gb117a47afalseOK43394342mirror.2smithi13219.0.0-1979-gb117a47afalseOKtestd6055756-c87a-420f-9f23-02c2bbaa6857up+replayingreplaying, {"bytes_per_second":17685504.0,"bytes_per_snapshot":533669376.0,"last_snapshot_bytes":530565120,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1710197483,"remote_snapshot_timestamp":1710197483,"replay_state":"idle"}42114267mirror.0smithi1322024-03-11 22:51:39cluster2baf1db9c-11db-489a-a01f-16a50bade1d5up+stoppedlocal image is primary2024-03-11 22:51:39' 2024-03-11T22:51:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local last_update state 2024-03-11T22:51:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-11T22:51:44.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ last_update='2024-03-11 22:51:39' 2024-03-11T22:51:44.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-11T22:51:44.957 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ state=up+replaying 2024-03-11T22:51:44.957 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo up+replaying 2024-03-11T22:51:44.957 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep up+replaying 2024-03-11T22:51:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:up+replaying 2024-03-11T22:51:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:51:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:51:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap3 2024-03-11T22:51:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror test snap3 2024-03-11T22:51:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:51:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:51:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:51:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap3 2024-03-11T22:51:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2024-03-11T22:52:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-11T22:52:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:52:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:52:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:52:13.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-11T22:52:13.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:52:13.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:52:13.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:52:13.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T22:52:13.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T22:52:13.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:52:13.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T22:52:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-11T22:52:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:52:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:52:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:52:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T22:52:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T22:52:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T22:52:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-11T22:52:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T22:52:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T22:52:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:52:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T22:52:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T22:52:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T22:52:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:52:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T22:52:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T22:52:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T22:52:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:52:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-11T22:52:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T22:52:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:52:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:52:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T22:52:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T22:52:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:52:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:52:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:52:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T22:52:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T22:52:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:52:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:52:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T22:52:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:52:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T22:52:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T22:52:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:52:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:52:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T22:52:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T22:52:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:52:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:52:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:52:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:52:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T22:52:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:52:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:52:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T22:52:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T22:52:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T22:52:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:52:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:52:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:52:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-11T22:52:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T22:52:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T22:52:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T22:52:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T22:52:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T22:52:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T22:52:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T22:52:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:52:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-11T22:52:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T22:52:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:52:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:52:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:52:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T22:52:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T22:52:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-03-11T22:52:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T22:52:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:52:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:52:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:52:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-test.status 2024-03-11T22:52:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-test.status 2024-03-11T22:52:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T22:52:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.status 2024-03-11T22:52:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-test.status 2024-03-11T22:52:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-test.status 2024-03-11T22:52:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T22:52:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:52:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-03-11T22:52:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:52:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:52:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:52:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-03-11T22:52:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 11 2024-03-11T22:52:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.status 2024-03-11T22:52:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:52:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:52:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:52:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.status 2024-03-11T22:52:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-03-11T22:52:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:52:17.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T22:52:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=11 2024-03-11T22:52:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:52:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:52:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T22:52:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:52:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:52:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:52:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:52:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:52:17.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:52:17.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:52:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:52:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:52:17.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T22:52:17.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:52:17.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T22:52:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T22:52:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 11 = '' 2024-03-11T22:52:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:52:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T22:52:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:52:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:52:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:52:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:52:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:52:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:52:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:52:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:52:17.787 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:52:17.788 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T22:52:17.788 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:52:17.788 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T22:52:17.788 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T22:52:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 11 = '' 2024-03-11T22:52:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:52:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T22:52:18.590 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:52:18.590 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:52:18.590 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:52:18.590 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:52:18.591 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:52:18.591 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:52:18.591 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:52:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:52:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:52:18.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T22:52:18.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:52:18.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T22:52:18.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T22:52:18.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 11 = '' 2024-03-11T22:52:18.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:52:18.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T22:52:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:52:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:52:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:52:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:52:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:52:20.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:52:20.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:52:20.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:52:20.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=11 2024-03-11T22:52:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 11 = 11 2024-03-11T22:52:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:52:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2024-03-11T22:52:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:52:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:52:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:52:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap3 2024-03-11T22:52:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T22:52:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T22:52:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T22:52:21.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2024-03-11T22:52:21.618 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'test': 2024-03-11T22:52:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 512 MiB in 128 objects 2024-03-11T22:52:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T22:52:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 6 2024-03-11T22:52:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 10ab50368aa4 2024-03-11T22:52:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.10ab50368aa4 2024-03-11T22:52:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T22:52:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T22:52:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T22:52:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T22:52:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 22:49:42 2024 2024-03-11T22:52:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 22:51:29 2024 2024-03-11T22:52:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 22:49:42 2024 2024-03-11T22:52:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T22:52:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T22:52:21.620 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T22:52:21.620 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: d6055756-c87a-420f-9f23-02c2bbaa6857 2024-03-11T22:52:21.620 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T22:52:21.623 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:52:21.623 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snap2-clean ']' 2024-03-11T22:52:21.623 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap2-clean 2024-03-11T22:52:21.623 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:52:21.623 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:52:21.623 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap2-clean 2024-03-11T22:52:21.623 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:52:21.623 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:52:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:52:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:52:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:52:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:52:22.253 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:52:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:52:23.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:52:23.220 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:52:23.335 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:52:23.335 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap3 2024-03-11T22:52:23.335 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:52:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:52:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap3 2024-03-11T22:52:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:52:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:52:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:52:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:52:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:52:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:52:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:52:24.638 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:52:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:52:24.865 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:52:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:52:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ clean_snap_name=snap3-clean 2024-03-11T22:52:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror test snap3-clean 2024-03-11T22:52:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:52:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:52:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:52:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap3-clean 2024-03-11T22:52:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2024-03-11T22:52:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T22:52:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror test 2024-03-11T22:52:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:52:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:52:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:52:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T22:52:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=6 2024-03-11T22:52:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T22:52:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-03-11T22:52:25.456 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 86620160 bytes 2024-03-11T22:52:25.597 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 115559424 bytes 2024-03-11T22:52:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 143785984 bytes 2024-03-11T22:52:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 168960000 bytes 2024-03-11T22:52:25.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 226484224 bytes 2024-03-11T22:52:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 258329600 bytes 2024-03-11T22:52:26.207 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 266518528 bytes 2024-03-11T22:52:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 300497920 bytes 2024-03-11T22:52:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 345627648 bytes 2024-03-11T22:52:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 365241344 bytes 2024-03-11T22:52:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 423230464 bytes 2024-03-11T22:52:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 480205824 bytes 2024-03-11T22:52:27.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 535289856 bytes 2024-03-11T22:52:27.321 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 583704576 bytes 2024-03-11T22:52:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 602869760 bytes 2024-03-11T22:52:27.519 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 618827776 bytes 2024-03-11T22:52:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 659169280 bytes 2024-03-11T22:52:27.941 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 763218944 bytes 2024-03-11T22:52:28.036 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 766891008 bytes 2024-03-11T22:52:28.209 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 813266944 bytes 2024-03-11T22:52:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 825864192 bytes 2024-03-11T22:52:28.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 826461184 bytes 2024-03-11T22:52:28.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 945709056 bytes 2024-03-11T22:52:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 981115904 bytes 2024-03-11T22:52:29.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1025970176 bytes 2024-03-11T22:52:29.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1123903488 bytes 2024-03-11T22:52:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1143524352 bytes 2024-03-11T22:52:29.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1203989504 bytes 2024-03-11T22:52:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1237695488 bytes 2024-03-11T22:52:29.860 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1288439808 bytes 2024-03-11T22:52:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1289079808 bytes 2024-03-11T22:52:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1345094656 bytes 2024-03-11T22:52:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1363169280 bytes 2024-03-11T22:52:30.416 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1432287232 bytes 2024-03-11T22:52:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1481459712 bytes 2024-03-11T22:52:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1482058752 bytes 2024-03-11T22:52:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1519203328 bytes 2024-03-11T22:52:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1527260160 bytes 2024-03-11T22:52:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1543556096 bytes 2024-03-11T22:52:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T22:52:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T22:52:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T22:52:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:52:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-11T22:52:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:52:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:52:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:52:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:52:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local description_pattern= 2024-03-11T22:52:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local service_pattern= 2024-03-11T22:52:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:52:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T22:52:32.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-11T22:52:32.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:52:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:52:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:52:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:52:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local description_pattern= 2024-03-11T22:52:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local service_pattern= 2024-03-11T22:52:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status 2024-03-11T22:52:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-11T22:52:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-11T22:52:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status='test: 2024-03-11T22:52:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: global_id: d6055756-c87a-420f-9f23-02c2bbaa6857 2024-03-11T22:52:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: state: up+replaying 2024-03-11T22:52:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":533669376.0,"last_snapshot_bytes":530565120,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1710197483,"remote_snapshot_timestamp":1710197483,"replay_state":"idle"} 2024-03-11T22:52:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: service: mirror.0 on smithi132 2024-03-11T22:52:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: last_update: 2024-03-11 22:52:09 2024-03-11T22:52:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: peer_sites: 2024-03-11T22:52:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: name: cluster2 2024-03-11T22:52:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: state: up+stopped 2024-03-11T22:52:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: description: local image is primary 2024-03-11T22:52:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: last_update: 2024-03-11 22:52:09' 2024-03-11T22:52:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^ state: .*up+replaying' 2024-03-11T22:52:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: state: up+replaying 2024-03-11T22:52:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^ description: .*' 2024-03-11T22:52:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":533669376.0,"last_snapshot_bytes":530565120,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1710197483,"remote_snapshot_timestamp":1710197483,"replay_state":"idle"} 2024-03-11T22:52:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T22:52:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo up+replaying 2024-03-11T22:52:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^up+' 2024-03-11T22:52:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:up+replaying 2024-03-11T22:52:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep 'service: *mirror..* on ' 2024-03-11T22:52:32.175 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: service: mirror.0 on smithi132 2024-03-11T22:52:32.176 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local last_update 2024-03-11T22:52:32.176 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-11T22:52:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ last_update='2024-03-11 22:52:09' 2024-03-11T22:52:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-11 22:52:09' 2024-03-11T22:52:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:52:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:52:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:52:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:52:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local 'prev_last_update=2024-03-11 22:52:09' 2024-03-11T22:52:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status 2024-03-11T22:52:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-11T22:52:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-11T22:52:32.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status='OKOKOK142104270mirror.1smithi13219.0.0-1979-gb117a47afalseOK42114267mirror.0smithi13219.0.0-1979-gb117a47atrueOK42174275mirror.3smithi13219.0.0-1979-gb117a47afalseOK43394342mirror.2smithi13219.0.0-1979-gb117a47afalseOKtestd6055756-c87a-420f-9f23-02c2bbaa6857up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":533669376.0,"last_snapshot_bytes":530565120,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1710197483,"remote_snapshot_timestamp":1710197483,"replay_state":"idle"}42114267mirror.0smithi1322024-03-11 22:52:09cluster2baf1db9c-11db-489a-a01f-16a50bade1d5up+stoppedlocal image is primary2024-03-11 22:52:09' 2024-03-11T22:52:32.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local last_update state 2024-03-11T22:52:32.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-11T22:52:32.239 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ last_update='2024-03-11 22:52:09' 2024-03-11T22:52:32.239 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-11T22:52:32.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ state=up+replaying 2024-03-11T22:52:32.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo up+replaying 2024-03-11T22:52:32.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep up+replaying 2024-03-11T22:52:32.242 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:up+replaying 2024-03-11T22:52:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:52:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:52:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap4 2024-03-11T22:52:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror test snap4 2024-03-11T22:52:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:52:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:52:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:52:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap4 2024-03-11T22:52:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2024-03-11T22:52:32.657 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4247755250590467) > weight (0.25) 2024-03-11T22:52:32.658 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8855386945046625) > weight (0.25) 2024-03-11T22:52:32.658 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.456404515738024) > weight (0.25) 2024-03-11T22:52:32.658 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.5982308029605817) > weight (0.25) 2024-03-11T22:52:32.658 INFO:tasks.rbd_mirror_thrash:waiting for 72 secs before thrashing 2024-03-11T22:52:35.651 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.29287031218422643) > weight (0.25) 2024-03-11T22:52:35.652 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.33748818579735984) > weight (0.25) 2024-03-11T22:52:35.652 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.26363024404251145) > weight (0.25) 2024-03-11T22:52:35.652 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-03-11T22:52:35.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-03-11T22:52:35.652 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-03-11T22:52:35.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi132.stderr:2024-03-11T22:52:35.648+0000 7f4bf9f26640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 40164) UID: 1000 2024-03-11T22:52:38.653 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-03-11T22:52:38.654 INFO:teuthology.orchestra.run:waiting for 600 2024-03-11T22:52:38.654 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-03-11T22:52:38.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-03-11T22:52:38.655 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-03-11T22:52:38.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-03-11T22:52:38.658 INFO:tasks.rbd_mirror_thrash:waiting for 112 secs before thrashing 2024-03-11T22:53:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-11T22:53:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:53:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:53:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:53:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-11T22:53:00.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:53:00.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:53:00.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:53:00.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T22:53:00.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T22:53:00.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:53:00.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T22:53:01.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-11T22:53:01.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:53:01.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:53:01.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:53:01.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T22:53:01.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T22:53:01.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T22:53:01.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-11T22:53:01.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T22:53:01.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:53:01.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T22:53:01.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T22:53:01.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T22:53:01.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T22:53:01.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:53:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T22:53:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T22:53:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T22:53:01.663 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:53:01.663 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-11T22:53:01.663 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T22:53:01.663 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:53:01.663 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:53:01.663 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T22:53:01.663 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T22:53:01.663 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:53:01.663 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:53:01.663 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:53:01.663 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T22:53:01.663 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T22:53:01.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:53:01.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:53:01.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T22:53:01.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:53:01.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T22:53:01.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T22:53:01.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:53:01.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:53:01.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T22:53:01.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T22:53:01.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:53:01.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:53:01.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:53:01.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:53:01.665 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T22:53:01.665 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:53:01.665 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:53:01.665 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T22:53:01.665 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T22:53:01.665 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T22:53:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:53:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:53:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:53:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-11T22:53:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T22:53:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T22:53:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T22:53:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T22:53:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T22:53:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T22:53:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T22:53:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:53:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-11T22:53:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T22:53:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:53:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:53:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:53:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T22:53:01.774 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T22:53:01.774 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-03-11T22:53:01.774 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T22:53:01.774 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:53:01.774 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:53:01.774 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:53:01.774 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-test.status 2024-03-11T22:53:01.774 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-test.status 2024-03-11T22:53:01.774 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T22:53:01.775 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.status 2024-03-11T22:53:01.775 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-test.status 2024-03-11T22:53:01.775 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-test.status 2024-03-11T22:53:01.776 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T22:53:01.776 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:53:01.776 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-03-11T22:53:01.776 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:53:01.776 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:53:01.776 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:53:01.776 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-03-11T22:53:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 14 2024-03-11T22:53:03.690 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.status 2024-03-11T22:53:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:53:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:53:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:53:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.status 2024-03-11T22:53:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:53:03.694 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-03-11T22:53:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T22:53:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=14 2024-03-11T22:53:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:53:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:53:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T22:53:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:53:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:53:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:53:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:53:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:53:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:53:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:53:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:53:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:53:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T22:53:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:53:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T22:53:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T22:53:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 14 = '' 2024-03-11T22:53:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:53:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T22:53:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:53:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:53:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:53:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:53:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:53:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:53:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:53:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:53:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:53:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T22:53:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:53:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T22:53:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T22:53:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 14 = '' 2024-03-11T22:53:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:53:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T22:53:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:53:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:53:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:53:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:53:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:53:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:53:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:53:05.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:53:05.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:53:05.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T22:53:05.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:53:05.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T22:53:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T22:53:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 14 = '' 2024-03-11T22:53:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:53:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T22:53:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:53:06.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:53:06.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:53:06.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:53:06.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:53:06.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:53:06.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:53:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:53:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=14 2024-03-11T22:53:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 14 = 14 2024-03-11T22:53:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:53:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2024-03-11T22:53:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:53:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:53:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:53:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap4 2024-03-11T22:53:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T22:53:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T22:53:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T22:53:07.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2024-03-11T22:53:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'test': 2024-03-11T22:53:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 512 MiB in 128 objects 2024-03-11T22:53:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T22:53:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 8 2024-03-11T22:53:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 10ab50368aa4 2024-03-11T22:53:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.10ab50368aa4 2024-03-11T22:53:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T22:53:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T22:53:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T22:53:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T22:53:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 22:49:42 2024 2024-03-11T22:53:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 22:51:29 2024 2024-03-11T22:53:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 22:49:42 2024 2024-03-11T22:53:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T22:53:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T22:53:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T22:53:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: d6055756-c87a-420f-9f23-02c2bbaa6857 2024-03-11T22:53:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T22:53:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:53:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snap3-clean ']' 2024-03-11T22:53:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap3-clean 2024-03-11T22:53:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:53:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:53:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap3-clean 2024-03-11T22:53:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:53:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:53:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:53:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:53:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:53:08.655 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:53:08.665 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:53:09.359 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:53:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:53:09.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:53:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:53:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap4 2024-03-11T22:53:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:53:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:53:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap4 2024-03-11T22:53:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:53:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:53:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:53:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:53:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:53:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:53:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:53:11.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:53:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:53:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:53:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:53:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ clean_snap_name=snap4-clean 2024-03-11T22:53:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror test snap4-clean 2024-03-11T22:53:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:53:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:53:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:53:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap4-clean 2024-03-11T22:53:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2024-03-11T22:53:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T22:53:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror test 2024-03-11T22:53:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:53:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:53:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:53:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T22:53:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=14 2024-03-11T22:53:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T22:53:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-03-11T22:53:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 52953088 bytes 2024-03-11T22:53:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 96819200 bytes 2024-03-11T22:53:12.478 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 167522304 bytes 2024-03-11T22:53:12.630 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 190129152 bytes 2024-03-11T22:53:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 245823488 bytes 2024-03-11T22:53:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 250865664 bytes 2024-03-11T22:53:13.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 310963200 bytes 2024-03-11T22:53:13.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 328265728 bytes 2024-03-11T22:53:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 339262464 bytes 2024-03-11T22:53:13.629 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 374763520 bytes 2024-03-11T22:53:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 422750208 bytes 2024-03-11T22:53:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 483087360 bytes 2024-03-11T22:53:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 498042880 bytes 2024-03-11T22:53:14.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 518340608 bytes 2024-03-11T22:53:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 532315136 bytes 2024-03-11T22:53:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 570987520 bytes 2024-03-11T22:53:14.628 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 622838784 bytes 2024-03-11T22:53:14.791 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 666562560 bytes 2024-03-11T22:53:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 707491840 bytes 2024-03-11T22:53:15.158 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 779121664 bytes 2024-03-11T22:53:15.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 835649536 bytes 2024-03-11T22:53:15.470 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 874632192 bytes 2024-03-11T22:53:15.655 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 918666240 bytes 2024-03-11T22:53:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 954876928 bytes 2024-03-11T22:53:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 987904000 bytes 2024-03-11T22:53:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1022816256 bytes 2024-03-11T22:53:16.249 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1065553920 bytes 2024-03-11T22:53:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1130564608 bytes 2024-03-11T22:53:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1217842176 bytes 2024-03-11T22:53:16.772 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1220547584 bytes 2024-03-11T22:53:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1295440896 bytes 2024-03-11T22:53:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1352586240 bytes 2024-03-11T22:53:17.300 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1381901312 bytes 2024-03-11T22:53:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1459093504 bytes 2024-03-11T22:53:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1463181312 bytes 2024-03-11T22:53:17.772 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1509138432 bytes 2024-03-11T22:53:17.938 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1558159360 bytes 2024-03-11T22:53:18.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1558785024 bytes 2024-03-11T22:53:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1641412608 bytes 2024-03-11T22:53:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1705955328 bytes 2024-03-11T22:53:18.878 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1879596032 bytes 2024-03-11T22:53:18.999 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1906542592 bytes 2024-03-11T22:53:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1936375808 bytes 2024-03-11T22:53:19.300 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 1997793280 bytes 2024-03-11T22:53:19.395 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 2005943296 bytes 2024-03-11T22:53:19.526 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 2051636224 bytes 2024-03-11T22:53:19.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 2078508032 bytes 2024-03-11T22:53:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 2101964800 bytes 2024-03-11T22:53:19.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 2145277952 bytes 2024-03-11T22:53:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 2202365952 bytes 2024-03-11T22:53:20.274 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5100 IOs, 2227407872 bytes 2024-03-11T22:53:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5200 IOs, 2325668864 bytes 2024-03-11T22:53:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5300 IOs, 2339044352 bytes 2024-03-11T22:53:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5400 IOs, 2403824640 bytes 2024-03-11T22:53:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5500 IOs, 2492340224 bytes 2024-03-11T22:53:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5600 IOs, 2567002112 bytes 2024-03-11T22:53:21.622 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5700 IOs, 2669656064 bytes 2024-03-11T22:53:21.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5800 IOs, 2670270464 bytes 2024-03-11T22:53:21.805 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5900 IOs, 2679445504 bytes 2024-03-11T22:53:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6000 IOs, 2709558272 bytes 2024-03-11T22:53:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6100 IOs, 2720459776 bytes 2024-03-11T22:53:22.215 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6200 IOs, 2782686208 bytes 2024-03-11T22:53:22.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6300 IOs, 2833838080 bytes 2024-03-11T22:53:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6400 IOs, 2852684800 bytes 2024-03-11T22:53:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6500 IOs, 2882495488 bytes 2024-03-11T22:53:22.762 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6600 IOs, 2920070144 bytes 2024-03-11T22:53:22.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6700 IOs, 2974283776 bytes 2024-03-11T22:53:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6800 IOs, 3001137152 bytes 2024-03-11T22:53:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6900 IOs, 3032338432 bytes 2024-03-11T22:53:23.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7000 IOs, 3067933696 bytes 2024-03-11T22:53:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7100 IOs, 3126736896 bytes 2024-03-11T22:53:23.680 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7200 IOs, 3184398336 bytes 2024-03-11T22:53:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7300 IOs, 3243906048 bytes 2024-03-11T22:53:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7400 IOs, 3252902912 bytes 2024-03-11T22:53:24.220 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7500 IOs, 3332035584 bytes 2024-03-11T22:53:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7600 IOs, 3346253824 bytes 2024-03-11T22:53:24.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7700 IOs, 3367855104 bytes 2024-03-11T22:53:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7800 IOs, 3412421632 bytes 2024-03-11T22:53:24.825 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7900 IOs, 3483512832 bytes 2024-03-11T22:53:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:8000 IOs, 3518052352 bytes 2024-03-11T22:53:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:8100 IOs, 3561402368 bytes 2024-03-11T22:53:25.463 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:8200 IOs, 3633001472 bytes 2024-03-11T22:53:25.593 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:8300 IOs, 3661897728 bytes 2024-03-11T22:53:25.687 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T22:53:25.687 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T22:53:25.687 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T22:53:25.687 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:53:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-11T22:53:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:53:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:53:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:53:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:53:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local description_pattern= 2024-03-11T22:53:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local service_pattern= 2024-03-11T22:53:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:53:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T22:53:26.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-11T22:53:26.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:53:26.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:53:26.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:53:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:53:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local description_pattern= 2024-03-11T22:53:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local service_pattern= 2024-03-11T22:53:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status 2024-03-11T22:53:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-11T22:53:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-11T22:53:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status='test: 2024-03-11T22:53:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: global_id: d6055756-c87a-420f-9f23-02c2bbaa6857 2024-03-11T22:53:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: state: up+replaying 2024-03-11T22:53:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: description: replaying, {"bytes_per_second":16879547.73,"bytes_per_snapshot":525551104.0,"last_snapshot_bytes":518235136,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1710197582,"remote_snapshot_timestamp":1710197582,"replay_state":"idle"} 2024-03-11T22:53:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: service: mirror.0 on smithi132 2024-03-11T22:53:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: last_update: 2024-03-11 22:53:09 2024-03-11T22:53:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: peer_sites: 2024-03-11T22:53:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: name: cluster2 2024-03-11T22:53:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: state: up+stopped 2024-03-11T22:53:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: description: local image is primary 2024-03-11T22:53:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: last_update: 2024-03-11 22:53:09' 2024-03-11T22:53:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^ state: .*up+replaying' 2024-03-11T22:53:26.743 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: state: up+replaying 2024-03-11T22:53:26.743 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^ description: .*' 2024-03-11T22:53:26.743 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: description: replaying, {"bytes_per_second":16879547.73,"bytes_per_snapshot":525551104.0,"last_snapshot_bytes":518235136,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1710197582,"remote_snapshot_timestamp":1710197582,"replay_state":"idle"} 2024-03-11T22:53:26.744 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T22:53:26.744 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo up+replaying 2024-03-11T22:53:26.744 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^up+' 2024-03-11T22:53:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:up+replaying 2024-03-11T22:53:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep 'service: *mirror..* on ' 2024-03-11T22:53:26.746 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: service: mirror.0 on smithi132 2024-03-11T22:53:26.746 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local last_update 2024-03-11T22:53:26.746 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-11T22:53:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ last_update='2024-03-11 22:53:09' 2024-03-11T22:53:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-11 22:53:09' 2024-03-11T22:53:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:53:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:53:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:53:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:53:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local 'prev_last_update=2024-03-11 22:53:09' 2024-03-11T22:53:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status 2024-03-11T22:53:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-11T22:53:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-11T22:53:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status='OKOKOK142104270mirror.1smithi13219.0.0-1979-gb117a47afalseOK42114267mirror.0smithi13219.0.0-1979-gb117a47atrueOK42174275mirror.3smithi13219.0.0-1979-gb117a47afalseOK43394342mirror.2smithi13219.0.0-1979-gb117a47afalseOKtestd6055756-c87a-420f-9f23-02c2bbaa6857up+replayingreplaying, {"bytes_per_second":16879547.73,"bytes_per_snapshot":525551104.0,"last_snapshot_bytes":518235136,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1710197582,"remote_snapshot_timestamp":1710197582,"replay_state":"idle"}42114267mirror.0smithi1322024-03-11 22:53:09cluster2baf1db9c-11db-489a-a01f-16a50bade1d5up+stoppedlocal image is primary2024-03-11 22:53:09' 2024-03-11T22:53:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local last_update state 2024-03-11T22:53:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-11T22:53:26.808 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ last_update='2024-03-11 22:53:09' 2024-03-11T22:53:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-11T22:53:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ state=up+replaying 2024-03-11T22:53:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo up+replaying 2024-03-11T22:53:26.811 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep up+replaying 2024-03-11T22:53:26.811 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:up+replaying 2024-03-11T22:53:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:53:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:53:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap5 2024-03-11T22:53:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror test snap5 2024-03-11T22:53:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:53:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:53:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:53:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap5 2024-03-11T22:53:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2024-03-11T22:53:44.659 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.723032057328877) > weight (0.25) 2024-03-11T22:53:44.660 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6256199068615337) > weight (0.25) 2024-03-11T22:53:44.660 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.35711555737923184) > weight (0.25) 2024-03-11T22:53:44.660 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.9264998083285104) > weight (0.25) 2024-03-11T22:53:44.660 INFO:tasks.rbd_mirror_thrash:waiting for 105 secs before thrashing 2024-03-11T22:53:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-11T22:53:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:53:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:53:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:53:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-11T22:53:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:53:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:53:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:53:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T22:53:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T22:53:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:53:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T22:53:53.733 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-11T22:53:53.733 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:53:53.733 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:53:53.733 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:53:53.733 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T22:53:53.734 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T22:53:53.734 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T22:53:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-11T22:53:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T22:53:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T22:53:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:53:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T22:53:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T22:53:53.736 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T22:53:53.736 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:53:53.736 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T22:53:53.736 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T22:53:53.736 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T22:53:53.739 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:53:53.739 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-11T22:53:53.739 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T22:53:53.739 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:53:53.739 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:53:53.739 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T22:53:53.739 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T22:53:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:53:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:53:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:53:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T22:53:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T22:53:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:53:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:53:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T22:53:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:53:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T22:53:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T22:53:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:53:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:53:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T22:53:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T22:53:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:53:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:53:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:53:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:53:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T22:53:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:53:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:53:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T22:53:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T22:53:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T22:53:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:53:53.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:53:53.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:53:53.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-11T22:53:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T22:53:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T22:53:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T22:53:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T22:53:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T22:53:53.856 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T22:53:53.856 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T22:53:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:53:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-11T22:53:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T22:53:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:53:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:53:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:53:53.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T22:53:53.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T22:53:53.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-03-11T22:53:53.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T22:53:53.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:53:53.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:53:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:53:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-test.status 2024-03-11T22:53:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-test.status 2024-03-11T22:53:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T22:53:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.status 2024-03-11T22:53:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-test.status 2024-03-11T22:53:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-test.status 2024-03-11T22:53:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T22:53:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:53:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-03-11T22:53:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:53:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:53:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:53:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-03-11T22:53:55.748 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 17 2024-03-11T22:53:55.762 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.status 2024-03-11T22:53:55.762 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:53:55.762 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:53:55.762 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:53:55.762 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.status 2024-03-11T22:53:55.762 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-03-11T22:53:55.764 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:53:55.827 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T22:53:55.829 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=17 2024-03-11T22:53:55.829 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:53:55.829 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:53:55.829 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T22:53:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:53:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:53:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:53:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:53:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:53:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:53:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:53:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:53:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:53:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T22:53:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:53:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T22:53:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T22:53:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 17 = '' 2024-03-11T22:53:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:53:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T22:53:56.482 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:53:56.482 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:53:56.482 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:53:56.482 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:53:56.482 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:53:56.482 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:53:56.482 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:53:56.529 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:53:56.530 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:53:56.531 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T22:53:56.531 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:53:56.531 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T22:53:56.531 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T22:53:56.532 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 17 = '' 2024-03-11T22:53:56.532 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:53:56.532 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T22:53:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:53:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:53:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:53:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:53:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:53:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:53:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:53:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:53:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:53:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T22:53:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:53:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T22:53:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T22:53:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 17 = '' 2024-03-11T22:53:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:53:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T22:53:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:53:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:53:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:53:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:53:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:53:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:53:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:53:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:53:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=17 2024-03-11T22:53:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 17 = 17 2024-03-11T22:53:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:53:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2024-03-11T22:53:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:53:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:53:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:53:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap5 2024-03-11T22:53:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T22:53:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T22:53:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T22:54:00.031 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2024-03-11T22:54:00.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'test': 2024-03-11T22:54:00.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 512 MiB in 128 objects 2024-03-11T22:54:00.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T22:54:00.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 10 2024-03-11T22:54:00.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 10ab50368aa4 2024-03-11T22:54:00.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.10ab50368aa4 2024-03-11T22:54:00.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T22:54:00.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T22:54:00.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T22:54:00.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T22:54:00.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 22:49:42 2024 2024-03-11T22:54:00.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 22:53:08 2024 2024-03-11T22:54:00.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 22:49:42 2024 2024-03-11T22:54:00.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T22:54:00.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T22:54:00.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T22:54:00.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: d6055756-c87a-420f-9f23-02c2bbaa6857 2024-03-11T22:54:00.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T22:54:00.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:54:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snap4-clean ']' 2024-03-11T22:54:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap4-clean 2024-03-11T22:54:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:54:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:54:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap4-clean 2024-03-11T22:54:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:54:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:54:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:54:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:54:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:54:00.706 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:54:00.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:54:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:54:01.521 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:54:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:54:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:54:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap5 2024-03-11T22:54:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:54:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:54:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap5 2024-03-11T22:54:01.846 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:54:01.846 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:54:01.846 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:54:01.846 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:54:01.846 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:54:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:54:02.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:54:03.157 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:54:03.167 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:54:03.377 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:54:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:54:03.492 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ clean_snap_name=snap5-clean 2024-03-11T22:54:03.492 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror test snap5-clean 2024-03-11T22:54:03.492 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:54:03.492 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:54:03.492 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:54:03.492 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap5-clean 2024-03-11T22:54:03.492 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2024-03-11T22:54:03.764 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T22:54:03.764 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror test 2024-03-11T22:54:03.764 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:54:03.764 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:54:03.764 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:54:03.764 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T22:54:03.765 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=8 2024-03-11T22:54:03.765 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T22:54:03.765 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-03-11T22:54:03.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 23447552 bytes 2024-03-11T22:54:04.175 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 88051712 bytes 2024-03-11T22:54:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 117159936 bytes 2024-03-11T22:54:04.490 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 153723904 bytes 2024-03-11T22:54:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 232996864 bytes 2024-03-11T22:54:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 279109632 bytes 2024-03-11T22:54:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 304137216 bytes 2024-03-11T22:54:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 341292032 bytes 2024-03-11T22:54:05.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 400798720 bytes 2024-03-11T22:54:05.481 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 434272256 bytes 2024-03-11T22:54:05.678 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 488666112 bytes 2024-03-11T22:54:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 500520960 bytes 2024-03-11T22:54:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 515060736 bytes 2024-03-11T22:54:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 563118080 bytes 2024-03-11T22:54:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 627727360 bytes 2024-03-11T22:54:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 724456448 bytes 2024-03-11T22:54:06.680 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 767461376 bytes 2024-03-11T22:54:06.851 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 814203904 bytes 2024-03-11T22:54:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 848951296 bytes 2024-03-11T22:54:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 939138048 bytes 2024-03-11T22:54:07.526 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 1042993152 bytes 2024-03-11T22:54:07.635 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 1066718208 bytes 2024-03-11T22:54:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 1098732544 bytes 2024-03-11T22:54:07.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 1140246528 bytes 2024-03-11T22:54:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1177195520 bytes 2024-03-11T22:54:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1195512832 bytes 2024-03-11T22:54:08.377 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1258565632 bytes 2024-03-11T22:54:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1282717696 bytes 2024-03-11T22:54:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1348728832 bytes 2024-03-11T22:54:08.880 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1384364032 bytes 2024-03-11T22:54:08.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1404237824 bytes 2024-03-11T22:54:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1435928576 bytes 2024-03-11T22:54:09.269 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1477699584 bytes 2024-03-11T22:54:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1500274688 bytes 2024-03-11T22:54:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1523655680 bytes 2024-03-11T22:54:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1557617664 bytes 2024-03-11T22:54:09.803 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1586594816 bytes 2024-03-11T22:54:09.911 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1599431680 bytes 2024-03-11T22:54:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1680816128 bytes 2024-03-11T22:54:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1737449472 bytes 2024-03-11T22:54:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1778122752 bytes 2024-03-11T22:54:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1812308992 bytes 2024-03-11T22:54:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1822387200 bytes 2024-03-11T22:54:10.855 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 1860493312 bytes 2024-03-11T22:54:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 1917604864 bytes 2024-03-11T22:54:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 1918211072 bytes 2024-03-11T22:54:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 1943270400 bytes 2024-03-11T22:54:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 2001942528 bytes 2024-03-11T22:54:11.524 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 2018388992 bytes 2024-03-11T22:54:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 2053472256 bytes 2024-03-11T22:54:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T22:54:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T22:54:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T22:54:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:54:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-11T22:54:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:54:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:54:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:54:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:54:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local description_pattern= 2024-03-11T22:54:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local service_pattern= 2024-03-11T22:54:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:54:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T22:54:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-11T22:54:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:54:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:54:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:54:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:54:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local description_pattern= 2024-03-11T22:54:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local service_pattern= 2024-03-11T22:54:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status 2024-03-11T22:54:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-11T22:54:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-11T22:54:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status='test: 2024-03-11T22:54:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: global_id: d6055756-c87a-420f-9f23-02c2bbaa6857 2024-03-11T22:54:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: state: up+replaying 2024-03-11T22:54:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: description: replaying, {"bytes_per_second":17891669.33,"bytes_per_snapshot":527492608.0,"last_snapshot_bytes":536750080,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1710197634,"remote_snapshot_timestamp":1710197634,"replay_state":"idle"} 2024-03-11T22:54:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: service: mirror.0 on smithi132 2024-03-11T22:54:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: last_update: 2024-03-11 22:54:09 2024-03-11T22:54:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: peer_sites: 2024-03-11T22:54:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: name: cluster2 2024-03-11T22:54:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: state: up+stopped 2024-03-11T22:54:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: description: local image is primary 2024-03-11T22:54:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: last_update: 2024-03-11 22:54:09' 2024-03-11T22:54:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^ state: .*up+replaying' 2024-03-11T22:54:12.828 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: state: up+replaying 2024-03-11T22:54:12.828 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^ description: .*' 2024-03-11T22:54:12.828 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: description: replaying, {"bytes_per_second":17891669.33,"bytes_per_snapshot":527492608.0,"last_snapshot_bytes":536750080,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1710197634,"remote_snapshot_timestamp":1710197634,"replay_state":"idle"} 2024-03-11T22:54:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T22:54:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo up+replaying 2024-03-11T22:54:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^up+' 2024-03-11T22:54:12.830 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:up+replaying 2024-03-11T22:54:12.830 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep 'service: *mirror..* on ' 2024-03-11T22:54:12.831 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: service: mirror.0 on smithi132 2024-03-11T22:54:12.831 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local last_update 2024-03-11T22:54:12.831 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-11T22:54:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ last_update='2024-03-11 22:54:09' 2024-03-11T22:54:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-11 22:54:09' 2024-03-11T22:54:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:54:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:54:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:54:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:54:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local 'prev_last_update=2024-03-11 22:54:09' 2024-03-11T22:54:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status 2024-03-11T22:54:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-11T22:54:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-11T22:54:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status='OKOKOK142104270mirror.1smithi13219.0.0-1979-gb117a47afalseOK42114267mirror.0smithi13219.0.0-1979-gb117a47atrueOK42174275mirror.3smithi13219.0.0-1979-gb117a47afalseOK43394342mirror.2smithi13219.0.0-1979-gb117a47afalseOKtestd6055756-c87a-420f-9f23-02c2bbaa6857up+replayingreplaying, {"bytes_per_second":17891669.33,"bytes_per_snapshot":527492608.0,"last_snapshot_bytes":536750080,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1710197634,"remote_snapshot_timestamp":1710197634,"replay_state":"idle"}42114267mirror.0smithi1322024-03-11 22:54:09cluster2baf1db9c-11db-489a-a01f-16a50bade1d5up+stoppedlocal image is primary2024-03-11 22:54:09' 2024-03-11T22:54:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local last_update state 2024-03-11T22:54:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-11T22:54:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ last_update='2024-03-11 22:54:09' 2024-03-11T22:54:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-11T22:54:12.896 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ state=up+replaying 2024-03-11T22:54:12.896 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo up+replaying 2024-03-11T22:54:12.896 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep up+replaying 2024-03-11T22:54:12.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:up+replaying 2024-03-11T22:54:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:54:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:54:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap6 2024-03-11T22:54:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror test snap6 2024-03-11T22:54:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:54:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:54:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:54:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap6 2024-03-11T22:54:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2024-03-11T22:54:30.659 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-03-11T22:54:30.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-03-11T22:54:30.660 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-03-11T22:54:30.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr:2024-03-11T22:54:30.657+0000 7ff83bc78640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 40045) UID: 1000 2024-03-11T22:54:37.661 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-03-11T22:54:37.662 INFO:teuthology.orchestra.run:waiting for 600 2024-03-11T22:54:37.662 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-03-11T22:54:37.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-03-11T22:54:37.662 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-03-11T22:54:37.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-03-11T22:54:37.665 INFO:tasks.rbd_mirror_thrash:waiting for 113 secs before thrashing 2024-03-11T22:54:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-11T22:54:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:54:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:54:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:54:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-11T22:54:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:54:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:54:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:54:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T22:54:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T22:54:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:54:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T22:54:40.886 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-11T22:54:40.886 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:54:40.886 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:54:40.886 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:54:40.886 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T22:54:40.886 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T22:54:40.886 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T22:54:40.887 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-11T22:54:40.887 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T22:54:40.887 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T22:54:40.887 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:54:40.887 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T22:54:40.887 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T22:54:40.887 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T22:54:40.887 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:54:40.887 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T22:54:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T22:54:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T22:54:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:54:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-11T22:54:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T22:54:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:54:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:54:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T22:54:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T22:54:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:54:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:54:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:54:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T22:54:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T22:54:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:54:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:54:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T22:54:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:54:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T22:54:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T22:54:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:54:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:54:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T22:54:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T22:54:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:54:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:54:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:54:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:54:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T22:54:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:54:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:54:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T22:54:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T22:54:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T22:54:40.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:54:40.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:54:40.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:54:40.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-11T22:54:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T22:54:41.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T22:54:41.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T22:54:41.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T22:54:41.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T22:54:41.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T22:54:41.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T22:54:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:54:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-11T22:54:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T22:54:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:54:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:54:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:54:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T22:54:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T22:54:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-03-11T22:54:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T22:54:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:54:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:54:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:54:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-test.status 2024-03-11T22:54:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-test.status 2024-03-11T22:54:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T22:54:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.status 2024-03-11T22:54:41.006 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-test.status 2024-03-11T22:54:41.006 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-test.status 2024-03-11T22:54:41.006 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T22:54:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:54:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-03-11T22:54:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:54:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:54:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:54:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-03-11T22:54:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 20 2024-03-11T22:54:42.887 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.status 2024-03-11T22:54:42.888 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:54:42.888 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:54:42.888 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:54:42.888 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.status 2024-03-11T22:54:42.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-03-11T22:54:42.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:54:42.962 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T22:54:42.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=20 2024-03-11T22:54:42.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:54:42.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:54:42.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T22:54:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:54:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:54:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:54:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:54:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:54:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:54:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:54:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:54:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:54:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T22:54:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:54:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T22:54:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T22:54:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 20 = '' 2024-03-11T22:54:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:54:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T22:54:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:54:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:54:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:54:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:54:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:54:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:54:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:54:43.662 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:54:43.662 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:54:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T22:54:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:54:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T22:54:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T22:54:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 20 = '' 2024-03-11T22:54:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:54:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T22:54:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:54:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:54:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:54:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:54:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:54:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:54:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:54:44.517 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:54:44.517 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:54:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T22:54:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:54:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T22:54:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T22:54:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 20 = '' 2024-03-11T22:54:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:54:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T22:54:46.121 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:54:46.121 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:54:46.121 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:54:46.121 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:54:46.121 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:54:46.121 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:54:46.121 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:54:46.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:54:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=20 2024-03-11T22:54:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 20 = 20 2024-03-11T22:54:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:54:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2024-03-11T22:54:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:54:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:54:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:54:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap6 2024-03-11T22:54:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T22:54:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T22:54:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T22:54:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2024-03-11T22:54:47.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'test': 2024-03-11T22:54:47.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 512 MiB in 128 objects 2024-03-11T22:54:47.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T22:54:47.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 12 2024-03-11T22:54:47.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 10ab50368aa4 2024-03-11T22:54:47.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.10ab50368aa4 2024-03-11T22:54:47.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T22:54:47.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T22:54:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T22:54:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T22:54:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 22:49:42 2024 2024-03-11T22:54:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 22:53:08 2024 2024-03-11T22:54:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 22:49:42 2024 2024-03-11T22:54:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T22:54:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T22:54:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T22:54:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: d6055756-c87a-420f-9f23-02c2bbaa6857 2024-03-11T22:54:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T22:54:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:54:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snap5-clean ']' 2024-03-11T22:54:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap5-clean 2024-03-11T22:54:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:54:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:54:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap5-clean 2024-03-11T22:54:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:54:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:54:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:54:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:54:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:54:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:54:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:54:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:54:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:54:48.847 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:54:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:54:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap6 2024-03-11T22:54:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:54:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:54:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap6 2024-03-11T22:54:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:54:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:54:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:54:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:54:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:54:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:54:49.568 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:54:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:54:50.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:54:50.522 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:54:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:54:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ clean_snap_name=snap6-clean 2024-03-11T22:54:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror test snap6-clean 2024-03-11T22:54:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:54:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:54:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:54:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap6-clean 2024-03-11T22:54:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2024-03-11T22:54:51.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T22:54:51.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror test 2024-03-11T22:54:51.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:54:51.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:54:51.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:54:51.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T22:54:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=5 2024-03-11T22:54:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T22:54:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-03-11T22:54:51.424 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 49038336 bytes 2024-03-11T22:54:51.647 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 92203008 bytes 2024-03-11T22:54:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 110814208 bytes 2024-03-11T22:54:51.931 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 153162752 bytes 2024-03-11T22:54:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 289890304 bytes 2024-03-11T22:54:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 312339456 bytes 2024-03-11T22:54:52.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 404417536 bytes 2024-03-11T22:54:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 436228096 bytes 2024-03-11T22:54:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 469705728 bytes 2024-03-11T22:54:53.113 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 533695488 bytes 2024-03-11T22:54:53.254 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 569406464 bytes 2024-03-11T22:54:53.353 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 584732672 bytes 2024-03-11T22:54:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 680036352 bytes 2024-03-11T22:54:53.825 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 744075264 bytes 2024-03-11T22:54:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 818968576 bytes 2024-03-11T22:54:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 819595264 bytes 2024-03-11T22:54:54.181 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 823460864 bytes 2024-03-11T22:54:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 859372544 bytes 2024-03-11T22:54:54.456 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 896153600 bytes 2024-03-11T22:54:54.687 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 976499712 bytes 2024-03-11T22:54:54.808 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 1009720320 bytes 2024-03-11T22:54:55.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 1072515072 bytes 2024-03-11T22:54:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 1105843200 bytes 2024-03-11T22:54:55.342 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 1162667008 bytes 2024-03-11T22:54:55.447 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1186447360 bytes 2024-03-11T22:54:55.616 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1235641344 bytes 2024-03-11T22:54:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1275123712 bytes 2024-03-11T22:54:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1346767872 bytes 2024-03-11T22:54:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T22:54:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T22:54:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T22:54:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:54:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-11T22:54:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:54:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:54:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:54:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:54:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local description_pattern= 2024-03-11T22:54:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local service_pattern= 2024-03-11T22:54:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:54:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T22:54:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-11T22:54:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:54:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:54:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:54:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:54:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local description_pattern= 2024-03-11T22:54:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local service_pattern= 2024-03-11T22:54:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status 2024-03-11T22:54:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-11T22:54:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-11T22:54:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status='test: 2024-03-11T22:54:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: global_id: d6055756-c87a-420f-9f23-02c2bbaa6857 2024-03-11T22:54:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: state: up+replaying 2024-03-11T22:54:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":527492608.0,"last_snapshot_bytes":536750080,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1710197634,"remote_snapshot_timestamp":1710197634,"replay_state":"idle"} 2024-03-11T22:54:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: service: mirror.0 on smithi132 2024-03-11T22:54:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: last_update: 2024-03-11 22:54:39 2024-03-11T22:54:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: peer_sites: 2024-03-11T22:54:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: name: cluster2 2024-03-11T22:54:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: state: down+unknown 2024-03-11T22:54:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: description: status not found 2024-03-11T22:54:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: last_update: ' 2024-03-11T22:54:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^ state: .*up+replaying' 2024-03-11T22:54:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: state: up+replaying 2024-03-11T22:54:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^ description: .*' 2024-03-11T22:54:57.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":527492608.0,"last_snapshot_bytes":536750080,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1710197634,"remote_snapshot_timestamp":1710197634,"replay_state":"idle"} 2024-03-11T22:54:57.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T22:54:57.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo up+replaying 2024-03-11T22:54:57.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^up+' 2024-03-11T22:54:57.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:up+replaying 2024-03-11T22:54:57.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep 'service: *mirror..* on ' 2024-03-11T22:54:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: service: mirror.0 on smithi132 2024-03-11T22:54:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local last_update 2024-03-11T22:54:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-11T22:54:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ last_update='2024-03-11 22:54:39' 2024-03-11T22:54:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-11 22:54:39' 2024-03-11T22:54:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:54:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:54:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:54:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:54:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local 'prev_last_update=2024-03-11 22:54:39' 2024-03-11T22:54:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status 2024-03-11T22:54:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-11T22:54:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-11T22:54:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status='WARNINGOKWARNING142104270mirror.1smithi13219.0.0-1979-gb117a47afalseOK42114267mirror.0smithi13219.0.0-1979-gb117a47atrueOK42174275mirror.3smithi13219.0.0-1979-gb117a47afalseOK43394342mirror.2smithi13219.0.0-1979-gb117a47afalseOKtestd6055756-c87a-420f-9f23-02c2bbaa6857up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":527492608.0,"last_snapshot_bytes":536750080,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1710197634,"remote_snapshot_timestamp":1710197634,"replay_state":"idle"}42114267mirror.0smithi1322024-03-11 22:54:39cluster2baf1db9c-11db-489a-a01f-16a50bade1d5down+unknownstatus not found' 2024-03-11T22:54:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local last_update state 2024-03-11T22:54:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-11T22:54:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ last_update='2024-03-11 22:54:39' 2024-03-11T22:54:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-11T22:54:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ state=up+replaying 2024-03-11T22:54:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo up+replaying 2024-03-11T22:54:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep up+replaying 2024-03-11T22:54:57.278 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:up+replaying 2024-03-11T22:54:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:54:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:54:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap7 2024-03-11T22:54:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror test snap7 2024-03-11T22:54:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:54:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:54:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:54:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap7 2024-03-11T22:54:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2024-03-11T22:55:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-11T22:55:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:55:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:55:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:55:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-11T22:55:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:55:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:55:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:55:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T22:55:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T22:55:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:55:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T22:55:22.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-11T22:55:22.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:55:22.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:55:22.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:55:22.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T22:55:22.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T22:55:22.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T22:55:22.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-11T22:55:22.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T22:55:22.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T22:55:22.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:55:22.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T22:55:22.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T22:55:22.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T22:55:22.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:55:22.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T22:55:22.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T22:55:22.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T22:55:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:55:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-11T22:55:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T22:55:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:55:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:55:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T22:55:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T22:55:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:55:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:55:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:55:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T22:55:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T22:55:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:55:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:55:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T22:55:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:55:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T22:55:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T22:55:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:55:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:55:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T22:55:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T22:55:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:55:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:55:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:55:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:55:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T22:55:22.762 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:55:22.762 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:55:22.762 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T22:55:22.762 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T22:55:22.762 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T22:55:22.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:55:22.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:55:22.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:55:22.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-11T22:55:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T22:55:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T22:55:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T22:55:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T22:55:22.872 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T22:55:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T22:55:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T22:55:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:55:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-11T22:55:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T22:55:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:55:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:55:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:55:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T22:55:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T22:55:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-03-11T22:55:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T22:55:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:55:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:55:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:55:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-test.status 2024-03-11T22:55:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-test.status 2024-03-11T22:55:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T22:55:22.877 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.status 2024-03-11T22:55:22.877 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-test.status 2024-03-11T22:55:22.877 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-test.status 2024-03-11T22:55:22.877 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T22:55:22.877 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:55:22.877 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-03-11T22:55:22.877 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:55:22.877 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:55:22.877 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:55:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-03-11T22:55:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 23 2024-03-11T22:55:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.status 2024-03-11T22:55:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:55:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:55:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:55:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.status 2024-03-11T22:55:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:55:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-03-11T22:55:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T22:55:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=23 2024-03-11T22:55:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:55:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:55:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T22:55:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:55:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:55:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:55:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:55:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:55:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:55:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:55:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:55:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:55:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T22:55:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:55:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T22:55:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T22:55:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 23 = '' 2024-03-11T22:55:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:55:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T22:55:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:55:25.502 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:55:25.502 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:55:25.502 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:55:25.502 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:55:25.502 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:55:25.502 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:55:25.550 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:55:25.550 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:55:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T22:55:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:55:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T22:55:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T22:55:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 23 = '' 2024-03-11T22:55:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:55:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T22:55:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:55:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:55:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:55:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:55:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:55:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:55:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:55:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:55:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:55:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T22:55:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:55:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T22:55:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T22:55:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 23 = '' 2024-03-11T22:55:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:55:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T22:55:28.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:55:28.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:55:28.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:55:28.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:55:28.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:55:28.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:55:28.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:55:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:55:28.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=23 2024-03-11T22:55:28.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 23 = 23 2024-03-11T22:55:28.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:55:28.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2024-03-11T22:55:28.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:55:28.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:55:28.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:55:28.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap7 2024-03-11T22:55:28.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T22:55:28.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T22:55:28.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T22:55:29.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2024-03-11T22:55:29.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'test': 2024-03-11T22:55:29.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 512 MiB in 128 objects 2024-03-11T22:55:29.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T22:55:29.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 14 2024-03-11T22:55:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 10ab50368aa4 2024-03-11T22:55:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.10ab50368aa4 2024-03-11T22:55:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T22:55:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T22:55:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T22:55:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T22:55:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 22:49:42 2024 2024-03-11T22:55:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 22:54:47 2024 2024-03-11T22:55:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 22:49:42 2024 2024-03-11T22:55:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T22:55:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T22:55:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T22:55:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: d6055756-c87a-420f-9f23-02c2bbaa6857 2024-03-11T22:55:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T22:55:29.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:55:29.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snap6-clean ']' 2024-03-11T22:55:29.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap6-clean 2024-03-11T22:55:29.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:55:29.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:55:29.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap6-clean 2024-03-11T22:55:29.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:55:29.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:55:29.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:55:29.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:55:29.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:55:29.660 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-03-11T22:55:29.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-03-11T22:55:29.661 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-03-11T22:55:29.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi132.stderr:2024-03-11T22:55:29.657+0000 7f084e57d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 39975) UID: 1000 2024-03-11T22:55:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:55:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:55:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:55:30.438 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:55:30.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:55:30.764 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:55:30.764 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap7 2024-03-11T22:55:30.764 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:55:30.764 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:55:30.764 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap7 2024-03-11T22:55:30.764 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:55:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:55:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:55:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:55:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:55:31.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:55:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:55:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:55:32.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:55:32.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:55:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:55:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ clean_snap_name=snap7-clean 2024-03-11T22:55:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror test snap7-clean 2024-03-11T22:55:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:55:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:55:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:55:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap7-clean 2024-03-11T22:55:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2024-03-11T22:55:32.766 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T22:55:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror test 2024-03-11T22:55:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:55:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:55:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:55:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T22:55:32.768 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=14 2024-03-11T22:55:32.768 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T22:55:32.768 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-03-11T22:55:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 31464448 bytes 2024-03-11T22:55:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 54162432 bytes 2024-03-11T22:55:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 94485504 bytes 2024-03-11T22:55:33.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 97950720 bytes 2024-03-11T22:55:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 126891008 bytes 2024-03-11T22:55:33.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 146802688 bytes 2024-03-11T22:55:33.778 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 185031680 bytes 2024-03-11T22:55:33.888 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 202816512 bytes 2024-03-11T22:55:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 247396352 bytes 2024-03-11T22:55:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 328014848 bytes 2024-03-11T22:55:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 397554688 bytes 2024-03-11T22:55:34.662 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-03-11T22:55:34.662 INFO:teuthology.orchestra.run:waiting for 600 2024-03-11T22:55:34.662 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-03-11T22:55:34.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-03-11T22:55:34.663 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-03-11T22:55:34.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-03-11T22:55:34.666 INFO:tasks.rbd_mirror_thrash:waiting for 113 secs before thrashing 2024-03-11T22:55:34.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 404846592 bytes 2024-03-11T22:55:34.880 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 438972416 bytes 2024-03-11T22:55:35.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 476846080 bytes 2024-03-11T22:55:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 481294336 bytes 2024-03-11T22:55:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 525486080 bytes 2024-03-11T22:55:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 568446976 bytes 2024-03-11T22:55:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 628416512 bytes 2024-03-11T22:55:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 657096704 bytes 2024-03-11T22:55:35.887 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 674812928 bytes 2024-03-11T22:55:36.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 715908096 bytes 2024-03-11T22:55:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 747994112 bytes 2024-03-11T22:55:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 783302656 bytes 2024-03-11T22:55:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 816572416 bytes 2024-03-11T22:55:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 862988288 bytes 2024-03-11T22:55:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 888736768 bytes 2024-03-11T22:55:36.972 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 952814592 bytes 2024-03-11T22:55:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 967472128 bytes 2024-03-11T22:55:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 997435392 bytes 2024-03-11T22:55:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1030710272 bytes 2024-03-11T22:55:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1066729472 bytes 2024-03-11T22:55:37.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1088472064 bytes 2024-03-11T22:55:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1173431296 bytes 2024-03-11T22:55:38.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1239771136 bytes 2024-03-11T22:55:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1258237952 bytes 2024-03-11T22:55:38.300 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1295842304 bytes 2024-03-11T22:55:38.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1317262336 bytes 2024-03-11T22:55:38.550 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1341767680 bytes 2024-03-11T22:55:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1352634368 bytes 2024-03-11T22:55:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1418766336 bytes 2024-03-11T22:55:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1476950016 bytes 2024-03-11T22:55:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1511950336 bytes 2024-03-11T22:55:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1558051840 bytes 2024-03-11T22:55:39.522 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 1613010944 bytes 2024-03-11T22:55:39.642 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 1613625344 bytes 2024-03-11T22:55:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 1621334016 bytes 2024-03-11T22:55:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 1685780480 bytes 2024-03-11T22:55:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 1739026432 bytes 2024-03-11T22:55:40.301 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 1798130688 bytes 2024-03-11T22:55:40.479 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 1855809536 bytes 2024-03-11T22:55:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5100 IOs, 1941983232 bytes 2024-03-11T22:55:40.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5200 IOs, 1953102848 bytes 2024-03-11T22:55:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5300 IOs, 2036560896 bytes 2024-03-11T22:55:41.180 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5400 IOs, 2053131264 bytes 2024-03-11T22:55:41.299 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5500 IOs, 2075378688 bytes 2024-03-11T22:55:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5600 IOs, 2113488896 bytes 2024-03-11T22:55:41.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5700 IOs, 2144039936 bytes 2024-03-11T22:55:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5800 IOs, 2216265728 bytes 2024-03-11T22:55:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5900 IOs, 2263678976 bytes 2024-03-11T22:55:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6000 IOs, 2310688768 bytes 2024-03-11T22:55:42.299 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6100 IOs, 2331092992 bytes 2024-03-11T22:55:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6200 IOs, 2344791040 bytes 2024-03-11T22:55:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6300 IOs, 2402337792 bytes 2024-03-11T22:55:42.782 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6400 IOs, 2469142528 bytes 2024-03-11T22:55:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6500 IOs, 2476685312 bytes 2024-03-11T22:55:42.990 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6600 IOs, 2498543616 bytes 2024-03-11T22:55:43.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6700 IOs, 2524290048 bytes 2024-03-11T22:55:43.473 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6800 IOs, 2583653376 bytes 2024-03-11T22:55:43.586 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6900 IOs, 2584249344 bytes 2024-03-11T22:55:43.767 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7000 IOs, 2637970432 bytes 2024-03-11T22:55:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7100 IOs, 2679772160 bytes 2024-03-11T22:55:44.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7200 IOs, 2721153024 bytes 2024-03-11T22:55:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7300 IOs, 2798597120 bytes 2024-03-11T22:55:44.400 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7400 IOs, 2805786624 bytes 2024-03-11T22:55:44.546 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7500 IOs, 2837863424 bytes 2024-03-11T22:55:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7600 IOs, 2906688512 bytes 2024-03-11T22:55:44.909 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7700 IOs, 2943364096 bytes 2024-03-11T22:55:45.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7800 IOs, 2943998976 bytes 2024-03-11T22:55:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7900 IOs, 2989664256 bytes 2024-03-11T22:55:45.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:8000 IOs, 3022297088 bytes 2024-03-11T22:55:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:8100 IOs, 3038021632 bytes 2024-03-11T22:55:45.583 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:8200 IOs, 3087270912 bytes 2024-03-11T22:55:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:8300 IOs, 3109497856 bytes 2024-03-11T22:55:45.775 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:8400 IOs, 3110114304 bytes 2024-03-11T22:55:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:8500 IOs, 3168965632 bytes 2024-03-11T22:55:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:8600 IOs, 3198761984 bytes 2024-03-11T22:55:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:8700 IOs, 3220578304 bytes 2024-03-11T22:55:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:8800 IOs, 3268717568 bytes 2024-03-11T22:55:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:8900 IOs, 3307939840 bytes 2024-03-11T22:55:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:9000 IOs, 3383674880 bytes 2024-03-11T22:55:46.774 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T22:55:46.774 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T22:55:46.774 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T22:55:46.774 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:55:46.774 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-11T22:55:46.774 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:55:46.774 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:55:46.775 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:55:46.775 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:55:46.775 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local description_pattern= 2024-03-11T22:55:46.775 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local service_pattern= 2024-03-11T22:55:46.775 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:55:46.775 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T22:55:47.776 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-11T22:55:47.776 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:55:47.776 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:55:47.776 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:55:47.776 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:55:47.776 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local description_pattern= 2024-03-11T22:55:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local service_pattern= 2024-03-11T22:55:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status 2024-03-11T22:55:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-11T22:55:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-11T22:55:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status='test: 2024-03-11T22:55:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: global_id: d6055756-c87a-420f-9f23-02c2bbaa6857 2024-03-11T22:55:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: state: down+unknown 2024-03-11T22:55:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: description: status not found 2024-03-11T22:55:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: last_update: 2024-03-11T22:55:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: peer_sites: 2024-03-11T22:55:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: name: cluster2 2024-03-11T22:55:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: state: up+stopped 2024-03-11T22:55:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: description: local image is primary 2024-03-11T22:55:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: last_update: 2024-03-11 22:55:39' 2024-03-11T22:55:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^ state: .*up+replaying' 2024-03-11T22:55:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 1 2024-03-11T22:55:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:55:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 2 2024-03-11T22:55:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-11T22:55:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:55:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:55:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:55:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:55:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local description_pattern= 2024-03-11T22:55:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local service_pattern= 2024-03-11T22:55:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status 2024-03-11T22:55:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-11T22:55:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-11T22:55:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status='test: 2024-03-11T22:55:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: global_id: d6055756-c87a-420f-9f23-02c2bbaa6857 2024-03-11T22:55:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: state: down+unknown 2024-03-11T22:55:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: description: status not found 2024-03-11T22:55:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: last_update: 2024-03-11T22:55:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: peer_sites: 2024-03-11T22:55:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: name: cluster2 2024-03-11T22:55:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: state: up+stopped 2024-03-11T22:55:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: description: local image is primary 2024-03-11T22:55:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: last_update: 2024-03-11 22:55:39' 2024-03-11T22:55:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^ state: .*up+replaying' 2024-03-11T22:55:49.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 1 2024-03-11T22:55:49.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:55:49.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 4 2024-03-11T22:55:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-11T22:55:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:55:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:55:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:55:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:55:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local description_pattern= 2024-03-11T22:55:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local service_pattern= 2024-03-11T22:55:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status 2024-03-11T22:55:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-11T22:55:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-11T22:55:53.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status='test: 2024-03-11T22:55:53.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: global_id: d6055756-c87a-420f-9f23-02c2bbaa6857 2024-03-11T22:55:53.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: state: down+unknown 2024-03-11T22:55:53.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: description: status not found 2024-03-11T22:55:53.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: last_update: 2024-03-11T22:55:53.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: peer_sites: 2024-03-11T22:55:53.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: name: cluster2 2024-03-11T22:55:53.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: state: up+stopped 2024-03-11T22:55:53.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: description: local image is primary 2024-03-11T22:55:53.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: last_update: 2024-03-11 22:55:39' 2024-03-11T22:55:53.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^ state: .*up+replaying' 2024-03-11T22:55:53.950 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 1 2024-03-11T22:55:53.950 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:55:53.950 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 8 2024-03-11T22:56:01.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-11T22:56:01.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:56:01.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:56:01.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:56:01.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:56:01.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local description_pattern= 2024-03-11T22:56:01.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local service_pattern= 2024-03-11T22:56:01.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status 2024-03-11T22:56:01.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-11T22:56:01.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-11T22:56:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status='test: 2024-03-11T22:56:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: global_id: d6055756-c87a-420f-9f23-02c2bbaa6857 2024-03-11T22:56:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: state: down+unknown 2024-03-11T22:56:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: description: status not found 2024-03-11T22:56:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: last_update: 2024-03-11T22:56:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: peer_sites: 2024-03-11T22:56:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: name: cluster2 2024-03-11T22:56:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: state: up+stopped 2024-03-11T22:56:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: description: local image is primary 2024-03-11T22:56:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: last_update: 2024-03-11 22:55:39' 2024-03-11T22:56:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^ state: .*up+replaying' 2024-03-11T22:56:02.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 1 2024-03-11T22:56:02.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:56:02.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 8 2024-03-11T22:56:10.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-11T22:56:10.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:56:10.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:56:10.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:56:10.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:56:10.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local description_pattern= 2024-03-11T22:56:10.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local service_pattern= 2024-03-11T22:56:10.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status 2024-03-11T22:56:10.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-11T22:56:10.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-11T22:56:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status='test: 2024-03-11T22:56:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: global_id: d6055756-c87a-420f-9f23-02c2bbaa6857 2024-03-11T22:56:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: state: up+replaying 2024-03-11T22:56:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1710197723,"remote_snapshot_timestamp":1710197723,"replay_state":"idle"} 2024-03-11T22:56:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: service: mirror.1 on smithi132 2024-03-11T22:56:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: last_update: 2024-03-11 22:56:09 2024-03-11T22:56:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: peer_sites: 2024-03-11T22:56:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: name: cluster2 2024-03-11T22:56:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: state: up+stopped 2024-03-11T22:56:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: description: local image is primary 2024-03-11T22:56:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: last_update: 2024-03-11 22:56:09' 2024-03-11T22:56:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^ state: .*up+replaying' 2024-03-11T22:56:10.072 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: state: up+replaying 2024-03-11T22:56:10.072 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^ description: .*' 2024-03-11T22:56:10.072 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1710197723,"remote_snapshot_timestamp":1710197723,"replay_state":"idle"} 2024-03-11T22:56:10.073 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T22:56:10.073 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo up+replaying 2024-03-11T22:56:10.073 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^up+' 2024-03-11T22:56:10.074 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:up+replaying 2024-03-11T22:56:10.074 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep 'service: *mirror..* on ' 2024-03-11T22:56:10.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: service: mirror.1 on smithi132 2024-03-11T22:56:10.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local last_update 2024-03-11T22:56:10.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-11T22:56:10.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ last_update='2024-03-11 22:56:09' 2024-03-11T22:56:10.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-11 22:56:09' 2024-03-11T22:56:10.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:56:10.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:56:10.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:56:10.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:56:10.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local 'prev_last_update=2024-03-11 22:56:09' 2024-03-11T22:56:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status 2024-03-11T22:56:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-11T22:56:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-11T22:56:10.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status='OKOKOK142104270mirror.1smithi13219.0.0-1979-gb117a47afalseOK42174275mirror.3smithi13219.0.0-1979-gb117a47afalseOK43394342mirror.2smithi13219.0.0-1979-gb117a47atrueOK45164519mirror.0smithi13219.0.0-1979-gb117a47afalseOKtestd6055756-c87a-420f-9f23-02c2bbaa6857up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1710197723,"remote_snapshot_timestamp":1710197723,"replay_state":"idle"}42104270mirror.1smithi1322024-03-11 22:56:09cluster2baf1db9c-11db-489a-a01f-16a50bade1d5up+stoppedlocal image is primary2024-03-11 22:56:09' 2024-03-11T22:56:10.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local last_update state 2024-03-11T22:56:10.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-11T22:56:10.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ last_update='2024-03-11 22:56:09' 2024-03-11T22:56:10.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-11T22:56:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ state=up+replaying 2024-03-11T22:56:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo up+replaying 2024-03-11T22:56:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep up+replaying 2024-03-11T22:56:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:up+replaying 2024-03-11T22:56:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:56:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:56:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap8 2024-03-11T22:56:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror test snap8 2024-03-11T22:56:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:56:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:56:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:56:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap8 2024-03-11T22:56:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2024-03-11T22:56:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-11T22:56:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:56:14.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:56:14.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:56:14.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-11T22:56:14.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:56:14.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:56:14.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:56:14.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T22:56:14.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T22:56:14.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:56:14.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T22:56:15.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-11T22:56:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:56:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:56:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:56:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T22:56:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T22:56:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T22:56:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-11T22:56:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T22:56:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T22:56:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:56:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T22:56:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T22:56:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T22:56:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:56:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T22:56:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T22:56:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T22:56:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:56:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-11T22:56:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T22:56:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:56:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:56:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T22:56:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T22:56:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:56:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:56:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:56:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T22:56:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T22:56:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:56:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:56:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T22:56:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:56:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T22:56:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T22:56:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:56:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:56:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T22:56:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T22:56:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:56:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:56:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:56:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:56:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T22:56:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:56:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:56:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T22:56:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T22:56:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T22:56:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:56:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:56:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:56:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-11T22:56:15.815 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-11T22:56:15.815 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T22:56:15.815 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T22:56:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:56:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-03-11T22:56:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T22:56:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:56:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:56:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T22:56:15.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T22:56:15.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:56:15.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:56:15.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:56:15.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T22:56:15.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T22:56:15.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:56:15.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:56:15.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T22:56:15.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T22:56:15.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T22:56:15.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T22:56:15.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:56:15.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:56:15.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T22:56:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T22:56:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:56:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:56:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:56:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T22:56:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T22:56:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:56:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:56:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T22:56:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T22:56:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T22:56:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T22:56:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T22:56:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T22:56:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-03-11T22:56:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T22:56:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T22:56:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T22:56:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T22:56:15.919 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T22:56:15.921 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T22:56:15.921 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T22:56:15.921 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:56:15.921 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-11T22:56:15.921 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T22:56:15.921 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:56:15.921 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:56:15.921 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:56:15.921 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T22:56:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T22:56:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-03-11T22:56:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T22:56:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:56:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:56:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:56:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-test.status 2024-03-11T22:56:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-test.status 2024-03-11T22:56:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T22:56:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.status 2024-03-11T22:56:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-test.status 2024-03-11T22:56:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-test.status 2024-03-11T22:56:15.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T22:56:15.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:56:15.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-03-11T22:56:15.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:56:15.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:56:15.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:56:15.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-03-11T22:56:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 26 2024-03-11T22:56:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.status 2024-03-11T22:56:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:56:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:56:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:56:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.status 2024-03-11T22:56:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-03-11T22:56:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:56:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T22:56:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=26 2024-03-11T22:56:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:56:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:56:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T22:56:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:56:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:56:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:56:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:56:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:56:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:56:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:56:18.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:56:18.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:56:18.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T22:56:18.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:56:18.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T22:56:18.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T22:56:18.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 26 = '' 2024-03-11T22:56:18.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:56:18.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T22:56:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:56:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:56:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:56:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:56:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:56:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:56:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:56:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:56:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:56:18.515 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T22:56:18.515 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:56:18.515 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T22:56:18.515 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T22:56:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 26 = '' 2024-03-11T22:56:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:56:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T22:56:19.316 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:56:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:56:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:56:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:56:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:56:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:56:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:56:19.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:56:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:56:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T22:56:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:56:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T22:56:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T22:56:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 26 = '' 2024-03-11T22:56:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:56:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T22:56:21.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:56:21.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:56:21.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:56:21.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:56:21.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:56:21.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:56:21.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:56:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:56:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=26 2024-03-11T22:56:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 26 = 26 2024-03-11T22:56:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:56:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2024-03-11T22:56:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:56:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:56:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:56:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap8 2024-03-11T22:56:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T22:56:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T22:56:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T22:56:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2024-03-11T22:56:22.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'test': 2024-03-11T22:56:22.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 512 MiB in 128 objects 2024-03-11T22:56:22.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T22:56:22.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 16 2024-03-11T22:56:22.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 10ab50368aa4 2024-03-11T22:56:22.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.10ab50368aa4 2024-03-11T22:56:22.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T22:56:22.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T22:56:22.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T22:56:22.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T22:56:22.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 22:49:42 2024 2024-03-11T22:56:22.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 22:54:47 2024 2024-03-11T22:56:22.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 22:49:42 2024 2024-03-11T22:56:22.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T22:56:22.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T22:56:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T22:56:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: d6055756-c87a-420f-9f23-02c2bbaa6857 2024-03-11T22:56:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T22:56:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:56:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snap7-clean ']' 2024-03-11T22:56:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap7-clean 2024-03-11T22:56:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:56:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:56:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap7-clean 2024-03-11T22:56:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:56:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:56:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:56:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:56:22.110 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:56:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:56:22.737 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:56:23.525 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:56:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:56:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:56:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:56:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap8 2024-03-11T22:56:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:56:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:56:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap8 2024-03-11T22:56:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:56:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:56:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:56:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:56:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:56:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:56:24.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:56:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:56:25.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:56:25.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:56:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:56:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ clean_snap_name=snap8-clean 2024-03-11T22:56:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror test snap8-clean 2024-03-11T22:56:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:56:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:56:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:56:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap8-clean 2024-03-11T22:56:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2024-03-11T22:56:26.214 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T22:56:26.214 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror test 2024-03-11T22:56:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:56:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:56:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:56:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T22:56:26.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=13 2024-03-11T22:56:26.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T22:56:26.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-03-11T22:56:26.440 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 33845248 bytes 2024-03-11T22:56:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 54080512 bytes 2024-03-11T22:56:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 72377344 bytes 2024-03-11T22:56:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 134726656 bytes 2024-03-11T22:56:27.130 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 204431360 bytes 2024-03-11T22:56:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 268169216 bytes 2024-03-11T22:56:27.418 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 272535552 bytes 2024-03-11T22:56:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 319818752 bytes 2024-03-11T22:56:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 354410496 bytes 2024-03-11T22:56:27.869 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 381473792 bytes 2024-03-11T22:56:28.016 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 415719424 bytes 2024-03-11T22:56:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 417948672 bytes 2024-03-11T22:56:28.281 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 470489088 bytes 2024-03-11T22:56:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 498499584 bytes 2024-03-11T22:56:28.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 528863232 bytes 2024-03-11T22:56:28.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 545629184 bytes 2024-03-11T22:56:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 555456512 bytes 2024-03-11T22:56:28.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 603796480 bytes 2024-03-11T22:56:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 646557696 bytes 2024-03-11T22:56:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 725293056 bytes 2024-03-11T22:56:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 744692736 bytes 2024-03-11T22:56:29.660 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 817063936 bytes 2024-03-11T22:56:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 891554816 bytes 2024-03-11T22:56:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 905784320 bytes 2024-03-11T22:56:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 976488448 bytes 2024-03-11T22:56:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1023992832 bytes 2024-03-11T22:56:30.667 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-03-11T22:56:30.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-03-11T22:56:30.668 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-03-11T22:56:30.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr:2024-03-11T22:56:30.664+0000 7ff2f3d9d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 43977) UID: 1000 2024-03-11T22:56:30.690 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1136211968 bytes 2024-03-11T22:56:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1204603904 bytes 2024-03-11T22:56:31.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1223453696 bytes 2024-03-11T22:56:31.215 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1282008064 bytes 2024-03-11T22:56:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1298166784 bytes 2024-03-11T22:56:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1330448384 bytes 2024-03-11T22:56:31.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1393183744 bytes 2024-03-11T22:56:31.765 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1403311104 bytes 2024-03-11T22:56:31.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1473692672 bytes 2024-03-11T22:56:32.195 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1525474304 bytes 2024-03-11T22:56:32.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1553500160 bytes 2024-03-11T22:56:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1580261376 bytes 2024-03-11T22:56:32.557 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1593255936 bytes 2024-03-11T22:56:32.732 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1635122176 bytes 2024-03-11T22:56:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1681982464 bytes 2024-03-11T22:56:33.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1708642304 bytes 2024-03-11T22:56:33.275 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1795325952 bytes 2024-03-11T22:56:33.483 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 1864861696 bytes 2024-03-11T22:56:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 1909181440 bytes 2024-03-11T22:56:33.846 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 1945260032 bytes 2024-03-11T22:56:34.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 2025035776 bytes 2024-03-11T22:56:34.275 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 2074373120 bytes 2024-03-11T22:56:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 2075003904 bytes 2024-03-11T22:56:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 2159137792 bytes 2024-03-11T22:56:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5100 IOs, 2253726720 bytes 2024-03-11T22:56:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5200 IOs, 2306216960 bytes 2024-03-11T22:56:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5300 IOs, 2371918848 bytes 2024-03-11T22:56:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5400 IOs, 2392961024 bytes 2024-03-11T22:56:35.542 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5500 IOs, 2438597632 bytes 2024-03-11T22:56:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5600 IOs, 2442142720 bytes 2024-03-11T22:56:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5700 IOs, 2472312832 bytes 2024-03-11T22:56:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5800 IOs, 2499380224 bytes 2024-03-11T22:56:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5900 IOs, 2531749888 bytes 2024-03-11T22:56:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6000 IOs, 2539954176 bytes 2024-03-11T22:56:36.219 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6100 IOs, 2557049856 bytes 2024-03-11T22:56:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6200 IOs, 2603309056 bytes 2024-03-11T22:56:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6300 IOs, 2625558528 bytes 2024-03-11T22:56:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6400 IOs, 2711505920 bytes 2024-03-11T22:56:36.877 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6500 IOs, 2733539328 bytes 2024-03-11T22:56:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6600 IOs, 2818463744 bytes 2024-03-11T22:56:37.452 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6700 IOs, 2848720896 bytes 2024-03-11T22:56:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6800 IOs, 2856530944 bytes 2024-03-11T22:56:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6900 IOs, 2918885376 bytes 2024-03-11T22:56:37.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7000 IOs, 2997413888 bytes 2024-03-11T22:56:38.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7100 IOs, 3000768512 bytes 2024-03-11T22:56:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7200 IOs, 3109560320 bytes 2024-03-11T22:56:38.543 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7300 IOs, 3160698880 bytes 2024-03-11T22:56:38.695 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7400 IOs, 3197802496 bytes 2024-03-11T22:56:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7500 IOs, 3279385600 bytes 2024-03-11T22:56:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7600 IOs, 3319140352 bytes 2024-03-11T22:56:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7700 IOs, 3342216192 bytes 2024-03-11T22:56:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T22:56:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T22:56:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T22:56:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:56:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-11T22:56:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:56:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:56:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:56:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:56:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local description_pattern= 2024-03-11T22:56:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local service_pattern= 2024-03-11T22:56:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:56:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T22:56:39.668 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-03-11T22:56:39.668 INFO:teuthology.orchestra.run:waiting for 600 2024-03-11T22:56:39.669 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-03-11T22:56:39.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-03-11T22:56:39.669 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-03-11T22:56:39.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-03-11T22:56:39.672 INFO:tasks.rbd_mirror_thrash:waiting for 70 secs before thrashing 2024-03-11T22:56:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-11T22:56:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:56:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:56:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:56:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:56:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local description_pattern= 2024-03-11T22:56:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local service_pattern= 2024-03-11T22:56:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status 2024-03-11T22:56:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-11T22:56:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-11T22:56:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status='test: 2024-03-11T22:56:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: global_id: d6055756-c87a-420f-9f23-02c2bbaa6857 2024-03-11T22:56:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: state: up+replaying 2024-03-11T22:56:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: description: replaying, {"bytes_per_second":17463603.2,"bytes_per_snapshot":523908096.0,"last_snapshot_bytes":523908096,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1710197776,"remote_snapshot_timestamp":1710197776,"replay_state":"idle"} 2024-03-11T22:56:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: service: mirror.1 on smithi132 2024-03-11T22:56:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: last_update: 2024-03-11 22:56:39 2024-03-11T22:56:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: peer_sites: 2024-03-11T22:56:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: name: cluster2 2024-03-11T22:56:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: state: up+stopped 2024-03-11T22:56:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: description: local image is primary 2024-03-11T22:56:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: last_update: 2024-03-11 22:56:39' 2024-03-11T22:56:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^ state: .*up+replaying' 2024-03-11T22:56:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: state: up+replaying 2024-03-11T22:56:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^ description: .*' 2024-03-11T22:56:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: description: replaying, {"bytes_per_second":17463603.2,"bytes_per_snapshot":523908096.0,"last_snapshot_bytes":523908096,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1710197776,"remote_snapshot_timestamp":1710197776,"replay_state":"idle"} 2024-03-11T22:56:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T22:56:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo up+replaying 2024-03-11T22:56:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^up+' 2024-03-11T22:56:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:up+replaying 2024-03-11T22:56:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep 'service: *mirror..* on ' 2024-03-11T22:56:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: service: mirror.1 on smithi132 2024-03-11T22:56:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local last_update 2024-03-11T22:56:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-11T22:56:40.282 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ last_update='2024-03-11 22:56:39' 2024-03-11T22:56:40.282 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-11 22:56:39' 2024-03-11T22:56:40.282 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:56:40.282 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:56:40.282 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:56:40.282 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:56:40.282 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local 'prev_last_update=2024-03-11 22:56:39' 2024-03-11T22:56:40.282 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status 2024-03-11T22:56:40.282 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-11T22:56:40.282 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-11T22:56:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status='OKOKOK142104270mirror.1smithi13219.0.0-1979-gb117a47afalseOK42174275mirror.3smithi13219.0.0-1979-gb117a47afalseOK43394342mirror.2smithi13219.0.0-1979-gb117a47atrueOK45164519mirror.0smithi13219.0.0-1979-gb117a47afalseOKtestd6055756-c87a-420f-9f23-02c2bbaa6857up+replayingreplaying, {"bytes_per_second":17463603.2,"bytes_per_snapshot":523908096.0,"last_snapshot_bytes":523908096,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1710197776,"remote_snapshot_timestamp":1710197776,"replay_state":"idle"}42104270mirror.1smithi1322024-03-11 22:56:39cluster2baf1db9c-11db-489a-a01f-16a50bade1d5up+stoppedlocal image is primary2024-03-11 22:56:39' 2024-03-11T22:56:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local last_update state 2024-03-11T22:56:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-11T22:56:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ last_update='2024-03-11 22:56:39' 2024-03-11T22:56:40.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-11T22:56:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ state=up+replaying 2024-03-11T22:56:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo up+replaying 2024-03-11T22:56:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep up+replaying 2024-03-11T22:56:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:up+replaying 2024-03-11T22:56:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:56:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:56:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap9 2024-03-11T22:56:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror test snap9 2024-03-11T22:56:40.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:56:40.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:56:40.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:56:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap9 2024-03-11T22:56:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2024-03-11T22:57:08.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-11T22:57:08.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:57:08.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:57:08.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:57:08.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-11T22:57:08.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:57:08.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:57:08.834 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:57:08.834 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T22:57:08.834 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T22:57:08.834 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:57:08.834 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T22:57:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-11T22:57:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:57:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:57:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:57:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T22:57:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T22:57:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T22:57:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-11T22:57:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T22:57:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T22:57:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:57:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T22:57:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T22:57:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T22:57:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:57:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T22:57:09.839 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T22:57:09.839 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T22:57:09.839 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:57:09.839 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-11T22:57:09.839 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T22:57:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:57:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:57:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T22:57:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T22:57:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:57:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:57:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:57:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T22:57:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T22:57:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:57:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:57:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T22:57:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:57:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T22:57:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T22:57:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:57:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:57:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T22:57:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T22:57:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:57:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:57:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:57:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:57:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T22:57:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:57:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:57:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T22:57:09.842 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T22:57:09.842 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T22:57:09.859 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:57:09.859 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:57:09.860 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:57:09.860 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-11T22:57:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-11T22:57:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T22:57:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T22:57:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:57:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-03-11T22:57:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T22:57:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:57:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:57:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T22:57:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T22:57:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:57:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:57:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:57:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T22:57:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T22:57:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:57:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:57:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T22:57:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T22:57:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T22:57:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T22:57:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:57:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:57:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T22:57:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T22:57:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:57:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:57:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:57:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T22:57:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T22:57:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:57:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:57:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T22:57:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T22:57:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T22:57:09.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T22:57:09.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T22:57:09.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T22:57:09.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-03-11T22:57:10.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T22:57:10.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T22:57:10.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T22:57:10.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T22:57:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T22:57:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T22:57:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T22:57:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:57:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-11T22:57:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T22:57:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:57:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:57:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:57:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T22:57:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T22:57:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-03-11T22:57:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T22:57:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:57:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:57:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:57:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-test.status 2024-03-11T22:57:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-test.status 2024-03-11T22:57:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T22:57:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.status 2024-03-11T22:57:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-test.status 2024-03-11T22:57:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-test.status 2024-03-11T22:57:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T22:57:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:57:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-03-11T22:57:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:57:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:57:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:57:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-03-11T22:57:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 29 2024-03-11T22:57:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.status 2024-03-11T22:57:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:57:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:57:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:57:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.status 2024-03-11T22:57:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:57:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-03-11T22:57:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T22:57:11.936 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=29 2024-03-11T22:57:11.936 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:57:11.936 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:57:11.936 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T22:57:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:57:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:57:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:57:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:57:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:57:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:57:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:57:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:57:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:57:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T22:57:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:57:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T22:57:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T22:57:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 29 = '' 2024-03-11T22:57:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:57:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T22:57:12.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:57:12.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:57:12.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:57:12.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:57:12.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:57:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:57:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:57:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:57:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:57:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T22:57:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:57:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T22:57:12.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T22:57:12.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 29 = '' 2024-03-11T22:57:12.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:57:12.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T22:57:13.440 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:57:13.440 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:57:13.440 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:57:13.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:57:13.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:57:13.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:57:13.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:57:13.492 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:57:13.492 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:57:13.494 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T22:57:13.494 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:57:13.494 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T22:57:13.494 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T22:57:13.494 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 29 = '' 2024-03-11T22:57:13.494 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:57:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T22:57:15.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:57:15.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:57:15.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:57:15.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:57:15.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:57:15.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:57:15.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:57:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:57:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=29 2024-03-11T22:57:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 29 = 29 2024-03-11T22:57:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:57:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2024-03-11T22:57:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:57:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:57:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:57:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap9 2024-03-11T22:57:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T22:57:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T22:57:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T22:57:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2024-03-11T22:57:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'test': 2024-03-11T22:57:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 512 MiB in 128 objects 2024-03-11T22:57:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T22:57:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 18 2024-03-11T22:57:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 10ab50368aa4 2024-03-11T22:57:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.10ab50368aa4 2024-03-11T22:57:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T22:57:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T22:57:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T22:57:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T22:57:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 22:49:42 2024 2024-03-11T22:57:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 22:56:22 2024 2024-03-11T22:57:16.194 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 22:49:42 2024 2024-03-11T22:57:16.194 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T22:57:16.194 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T22:57:16.194 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T22:57:16.194 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: d6055756-c87a-420f-9f23-02c2bbaa6857 2024-03-11T22:57:16.194 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T22:57:16.198 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:57:16.198 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snap8-clean ']' 2024-03-11T22:57:16.198 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap8-clean 2024-03-11T22:57:16.198 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:57:16.198 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:57:16.198 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap8-clean 2024-03-11T22:57:16.198 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:57:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:57:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:57:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:57:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:57:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:57:16.791 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:57:17.526 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:57:17.536 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:57:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:57:17.864 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:57:17.864 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap9 2024-03-11T22:57:17.864 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:57:17.864 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:57:17.864 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap9 2024-03-11T22:57:17.864 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:57:17.864 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:57:17.864 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:57:17.864 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:57:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:57:18.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:57:18.482 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:57:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:57:19.252 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:57:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:57:19.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:57:19.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ clean_snap_name=snap9-clean 2024-03-11T22:57:19.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror test snap9-clean 2024-03-11T22:57:19.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:57:19.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:57:19.578 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:57:19.578 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap9-clean 2024-03-11T22:57:19.578 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2024-03-11T22:57:20.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T22:57:20.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror test 2024-03-11T22:57:20.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:57:20.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:57:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:57:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T22:57:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=5 2024-03-11T22:57:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T22:57:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-03-11T22:57:20.402 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 15128576 bytes 2024-03-11T22:57:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 29313024 bytes 2024-03-11T22:57:20.652 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 62935040 bytes 2024-03-11T22:57:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 82955264 bytes 2024-03-11T22:57:20.885 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 105443328 bytes 2024-03-11T22:57:20.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 113534976 bytes 2024-03-11T22:57:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 148931584 bytes 2024-03-11T22:57:21.306 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 203912192 bytes 2024-03-11T22:57:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 231071744 bytes 2024-03-11T22:57:21.571 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 279777280 bytes 2024-03-11T22:57:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 383174656 bytes 2024-03-11T22:57:22.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 397685760 bytes 2024-03-11T22:57:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 509719552 bytes 2024-03-11T22:57:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 530002944 bytes 2024-03-11T22:57:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 616284160 bytes 2024-03-11T22:57:22.941 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 720949248 bytes 2024-03-11T22:57:23.110 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 790397952 bytes 2024-03-11T22:57:23.278 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 814554112 bytes 2024-03-11T22:57:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 860511232 bytes 2024-03-11T22:57:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 875631616 bytes 2024-03-11T22:57:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 945177600 bytes 2024-03-11T22:57:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 1047356416 bytes 2024-03-11T22:57:24.195 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 1062262784 bytes 2024-03-11T22:57:24.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 1095495680 bytes 2024-03-11T22:57:24.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1129068544 bytes 2024-03-11T22:57:24.714 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1210547200 bytes 2024-03-11T22:57:24.876 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1252272128 bytes 2024-03-11T22:57:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1292796928 bytes 2024-03-11T22:57:25.129 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1300917248 bytes 2024-03-11T22:57:25.224 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T22:57:25.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T22:57:25.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T22:57:25.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:57:25.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-11T22:57:25.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:57:25.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:57:25.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:57:25.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:57:25.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local description_pattern= 2024-03-11T22:57:25.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local service_pattern= 2024-03-11T22:57:25.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:57:25.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T22:57:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-11T22:57:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:57:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:57:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:57:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:57:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local description_pattern= 2024-03-11T22:57:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local service_pattern= 2024-03-11T22:57:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status 2024-03-11T22:57:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-11T22:57:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-11T22:57:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status='test: 2024-03-11T22:57:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: global_id: d6055756-c87a-420f-9f23-02c2bbaa6857 2024-03-11T22:57:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: state: up+replaying 2024-03-11T22:57:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":523908096.0,"last_snapshot_bytes":523908096,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1710197776,"remote_snapshot_timestamp":1710197776,"replay_state":"idle"} 2024-03-11T22:57:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: service: mirror.1 on smithi132 2024-03-11T22:57:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: last_update: 2024-03-11 22:57:09 2024-03-11T22:57:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: peer_sites: 2024-03-11T22:57:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: name: cluster2 2024-03-11T22:57:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: state: up+stopped 2024-03-11T22:57:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: description: local image is primary 2024-03-11T22:57:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: last_update: 2024-03-11 22:57:09' 2024-03-11T22:57:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^ state: .*up+replaying' 2024-03-11T22:57:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: state: up+replaying 2024-03-11T22:57:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^ description: .*' 2024-03-11T22:57:26.287 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":523908096.0,"last_snapshot_bytes":523908096,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1710197776,"remote_snapshot_timestamp":1710197776,"replay_state":"idle"} 2024-03-11T22:57:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T22:57:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo up+replaying 2024-03-11T22:57:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep '^up+' 2024-03-11T22:57:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:up+replaying 2024-03-11T22:57:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep 'service: *mirror..* on ' 2024-03-11T22:57:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: service: mirror.1 on smithi132 2024-03-11T22:57:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local last_update 2024-03-11T22:57:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-11T22:57:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ last_update='2024-03-11 22:57:09' 2024-03-11T22:57:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-11 22:57:09' 2024-03-11T22:57:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:57:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:57:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:57:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state_pattern=up+replaying 2024-03-11T22:57:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local 'prev_last_update=2024-03-11 22:57:09' 2024-03-11T22:57:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status 2024-03-11T22:57:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-11T22:57:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-11T22:57:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status='OKOKOK142104270mirror.1smithi13219.0.0-1979-gb117a47afalseOK42174275mirror.3smithi13219.0.0-1979-gb117a47afalseOK43394342mirror.2smithi13219.0.0-1979-gb117a47atrueOK45164519mirror.0smithi13219.0.0-1979-gb117a47afalseOKtestd6055756-c87a-420f-9f23-02c2bbaa6857up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":523908096.0,"last_snapshot_bytes":523908096,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1710197776,"remote_snapshot_timestamp":1710197776,"replay_state":"idle"}42104270mirror.1smithi1322024-03-11 22:57:09cluster2baf1db9c-11db-489a-a01f-16a50bade1d5up+stoppedlocal image is primary2024-03-11 22:57:09' 2024-03-11T22:57:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local last_update state 2024-03-11T22:57:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-11T22:57:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ last_update='2024-03-11 22:57:09' 2024-03-11T22:57:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-11T22:57:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ state=up+replaying 2024-03-11T22:57:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo up+replaying 2024-03-11T22:57:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep up+replaying 2024-03-11T22:57:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:up+replaying 2024-03-11T22:57:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:57:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:57:26.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap10 2024-03-11T22:57:26.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror test snap10 2024-03-11T22:57:26.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:57:26.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:57:26.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:57:26.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap10 2024-03-11T22:57:26.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2024-03-11T22:57:27.666 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9446961395773211) > weight (0.25) 2024-03-11T22:57:27.666 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-03-11T22:57:27.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-03-11T22:57:27.667 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-03-11T22:57:27.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr:2024-03-11T22:57:27.664+0000 7fa466395640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 39992) UID: 1000 2024-03-11T22:57:32.668 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-03-11T22:57:32.668 INFO:teuthology.orchestra.run:waiting for 600 2024-03-11T22:57:32.669 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-03-11T22:57:32.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-03-11T22:57:32.669 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-03-11T22:57:32.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-03-11T22:57:32.672 INFO:tasks.rbd_mirror_thrash:waiting for 64 secs before thrashing 2024-03-11T22:57:49.673 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9082187757794339) > weight (0.25) 2024-03-11T22:57:49.673 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-03-11T22:57:49.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-03-11T22:57:49.674 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-03-11T22:57:49.674 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-03-11T22:57:49.674 INFO:teuthology.orchestra.run:waiting for 600 2024-03-11T22:57:49.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi132.stderr:2024-03-11T22:57:49.671+0000 7f0e40686640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 40063) UID: 1000 2024-03-11T22:57:54.742 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-11T22:57:54.742 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:57:54.742 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:57:54.743 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:57:54.743 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-11T22:57:54.743 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:57:54.743 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:57:54.743 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:57:54.743 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T22:57:54.743 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T22:57:54.743 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:57:54.743 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T22:57:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-11T22:57:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:57:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:57:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:57:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T22:57:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T22:57:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T22:57:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-11T22:57:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T22:57:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T22:57:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:57:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T22:57:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T22:57:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T22:57:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:57:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T22:57:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T22:57:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T22:57:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:57:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-11T22:57:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T22:57:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:57:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:57:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T22:57:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T22:57:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:57:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:57:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:57:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T22:57:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T22:57:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:57:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:57:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T22:57:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:57:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T22:57:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T22:57:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:57:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:57:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T22:57:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T22:57:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:57:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:57:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:57:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:57:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T22:57:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:57:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:57:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T22:57:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T22:57:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T22:57:55.769 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:57:55.769 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:57:55.769 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:57:55.769 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-11T22:57:55.776 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-03-11T22:57:55.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-03-11T22:57:55.776 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-03-11T22:57:55.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-03-11T22:57:55.779 INFO:tasks.rbd_mirror_thrash:waiting for 62 secs before thrashing 2024-03-11T22:57:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-11T22:57:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T22:57:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T22:57:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:57:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-03-11T22:57:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T22:57:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:57:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:57:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T22:57:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T22:57:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:57:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:57:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:57:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T22:57:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T22:57:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:57:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:57:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T22:57:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T22:57:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T22:57:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T22:57:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:57:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:57:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T22:57:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T22:57:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:57:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:57:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:57:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T22:57:55.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T22:57:55.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:57:55.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:57:55.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T22:57:55.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T22:57:55.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T22:57:55.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T22:57:55.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T22:57:55.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T22:57:55.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-03-11T22:57:55.957 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-11T22:57:55.957 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T22:57:55.957 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T22:57:55.960 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:57:55.960 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-03-11T22:57:55.960 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T22:57:55.960 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:57:55.960 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:57:55.960 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T22:57:55.960 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T22:57:55.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:57:55.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:57:55.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:57:55.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T22:57:55.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T22:57:55.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:57:55.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:57:55.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T22:57:55.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T22:57:55.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T22:57:55.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T22:57:55.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:57:55.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:57:55.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T22:57:55.962 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T22:57:55.962 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:57:55.962 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:57:55.962 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:57:55.962 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T22:57:55.962 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T22:57:55.962 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:57:55.962 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:57:55.962 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T22:57:55.962 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T22:57:55.962 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T22:57:55.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T22:57:55.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T22:57:55.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T22:57:55.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-03-11T22:57:56.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-11T22:57:56.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T22:57:56.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T22:57:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:57:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-03-11T22:57:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:3 2024-03-11T22:57:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:57:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:57:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-03-11T22:57:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:3 2024-03-11T22:57:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:57:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:57:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:57:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=3 2024-03-11T22:57:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 3 = cluster1:3 ']' 2024-03-11T22:57:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:57:56.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:57:56.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=3 2024-03-11T22:57:56.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T22:57:56.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-03-11T22:57:56.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:3 2024-03-11T22:57:56.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:57:56.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:57:56.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-03-11T22:57:56.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:3 2024-03-11T22:57:56.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:57:56.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:57:56.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:57:56.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T22:57:56.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 3 = cluster1:3 ']' 2024-03-11T22:57:56.058 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:57:56.058 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:57:56.058 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=3 2024-03-11T22:57:56.058 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=3 2024-03-11T22:57:56.058 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-03-11T22:57:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T22:57:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T22:57:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T22:57:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-03-11T22:57:56.148 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-11T22:57:56.148 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T22:57:56.148 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T22:57:56.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:57:56.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 1 2024-03-11T22:57:57.151 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T22:57:57.151 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T22:57:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:57:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-11T22:57:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T22:57:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:57:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:57:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T22:57:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T22:57:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:57:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:57:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:57:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T22:57:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T22:57:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:57:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:57:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T22:57:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:57:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T22:57:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T22:57:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:57:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:57:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T22:57:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T22:57:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:57:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:57:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:57:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:57:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T22:57:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:57:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:57:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T22:57:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T22:57:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T22:57:57.173 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:57:57.173 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:57:57.173 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:57:57.173 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-11T22:57:57.250 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-11T22:57:57.250 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T22:57:57.250 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T22:57:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:57:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-03-11T22:57:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T22:57:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:57:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:57:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T22:57:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T22:57:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:57:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:57:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:57:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T22:57:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T22:57:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:57:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:57:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T22:57:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T22:57:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T22:57:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T22:57:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:57:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:57:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T22:57:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T22:57:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:57:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:57:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:57:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T22:57:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T22:57:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:57:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:57:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T22:57:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T22:57:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T22:57:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T22:57:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T22:57:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T22:57:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-03-11T22:57:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-11T22:57:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T22:57:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T22:57:57.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:57:57.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-03-11T22:57:57.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T22:57:57.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:57:57.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:57:57.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T22:57:57.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T22:57:57.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:57:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:57:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:57:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T22:57:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T22:57:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:57:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:57:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T22:57:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T22:57:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T22:57:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T22:57:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:57:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:57:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T22:57:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T22:57:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:57:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:57:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:57:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T22:57:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T22:57:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:57:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:57:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T22:57:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T22:57:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T22:57:57.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T22:57:57.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T22:57:57.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T22:57:57.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-03-11T22:57:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-11T22:57:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T22:57:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T22:57:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:57:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-03-11T22:57:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:3 2024-03-11T22:57:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:57:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:57:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-03-11T22:57:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:3 2024-03-11T22:57:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:57:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:57:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:57:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=3 2024-03-11T22:57:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 3 = cluster1:3 ']' 2024-03-11T22:57:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:57:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:57:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=3 2024-03-11T22:57:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T22:57:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-03-11T22:57:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:3 2024-03-11T22:57:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:57:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:57:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-03-11T22:57:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:3 2024-03-11T22:57:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:57:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:57:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:57:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T22:57:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 3 = cluster1:3 ']' 2024-03-11T22:57:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:57:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:57:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=3 2024-03-11T22:57:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=3 2024-03-11T22:57:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-03-11T22:57:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T22:57:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T22:57:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T22:57:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-03-11T22:57:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-11T22:57:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T22:57:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T22:57:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:57:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 2 2024-03-11T22:57:59.535 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T22:57:59.535 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T22:57:59.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:57:59.539 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-11T22:57:59.539 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T22:57:59.539 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:57:59.539 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:57:59.539 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T22:57:59.539 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T22:57:59.539 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:57:59.539 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:57:59.539 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:57:59.539 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T22:57:59.539 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T22:57:59.539 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:57:59.539 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:57:59.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T22:57:59.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:57:59.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T22:57:59.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T22:57:59.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:57:59.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:57:59.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T22:57:59.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T22:57:59.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:57:59.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:57:59.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:57:59.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:57:59.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T22:57:59.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:57:59.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:57:59.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T22:57:59.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T22:57:59.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T22:57:59.559 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:57:59.559 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:57:59.560 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:57:59.560 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-11T22:57:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-11T22:57:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T22:57:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T22:57:59.642 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:57:59.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-03-11T22:57:59.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T22:57:59.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:57:59.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:57:59.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T22:57:59.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T22:57:59.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:57:59.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:57:59.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:57:59.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T22:57:59.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T22:57:59.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:57:59.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:57:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T22:57:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T22:57:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T22:57:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T22:57:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:57:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:57:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T22:57:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T22:57:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:57:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:57:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:57:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T22:57:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T22:57:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:57:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:57:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T22:57:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T22:57:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T22:57:59.660 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T22:57:59.660 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T22:57:59.660 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T22:57:59.660 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-03-11T22:57:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-11T22:57:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T22:57:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T22:57:59.738 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:57:59.738 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-03-11T22:57:59.738 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T22:57:59.738 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:57:59.738 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:57:59.738 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T22:57:59.738 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T22:57:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:57:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:57:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:57:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T22:57:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T22:57:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:57:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:57:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T22:57:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T22:57:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T22:57:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T22:57:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:57:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:57:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T22:57:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T22:57:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:57:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:57:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:57:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T22:57:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T22:57:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:57:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:57:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T22:57:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T22:57:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T22:57:59.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T22:57:59.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T22:57:59.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T22:57:59.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-03-11T22:57:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-11T22:57:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T22:57:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T22:57:59.838 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:57:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-03-11T22:57:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:3 2024-03-11T22:57:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:57:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:57:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-03-11T22:57:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:3 2024-03-11T22:57:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:57:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:57:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:57:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=3 2024-03-11T22:57:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 3 = cluster1:3 ']' 2024-03-11T22:57:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:57:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:57:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=3 2024-03-11T22:57:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T22:57:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-03-11T22:57:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:3 2024-03-11T22:57:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:57:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:57:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-03-11T22:57:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:3 2024-03-11T22:57:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:57:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:57:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:57:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T22:57:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 3 = cluster1:3 ']' 2024-03-11T22:57:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:57:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:57:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=3 2024-03-11T22:57:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=3 2024-03-11T22:57:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-03-11T22:57:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T22:57:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T22:57:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T22:57:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-03-11T22:57:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T22:57:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T22:57:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T22:57:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T22:57:59.936 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T22:57:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T22:57:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T22:57:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:57:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-11T22:57:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T22:57:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:57:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:57:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:57:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T22:57:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T22:57:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-03-11T22:57:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T22:57:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:57:59.939 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:57:59.939 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:57:59.939 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-test.status 2024-03-11T22:57:59.939 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-test.status 2024-03-11T22:57:59.939 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T22:57:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.status 2024-03-11T22:57:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-test.status 2024-03-11T22:57:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-test.status 2024-03-11T22:57:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T22:57:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:57:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-03-11T22:57:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:57:59.941 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:57:59.941 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:57:59.941 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-03-11T22:58:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 32 2024-03-11T22:58:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.status 2024-03-11T22:58:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:58:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.status 2024-03-11T22:58:01.774 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-03-11T22:58:01.774 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:58:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T22:58:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=32 2024-03-11T22:58:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:58:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:58:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T22:58:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:58:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:58:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:02.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:02.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:58:02.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:58:02.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:58:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:58:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:58:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T22:58:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:58:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T22:58:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T22:58:02.097 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 32 = '' 2024-03-11T22:58:02.097 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:58:02.097 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T22:58:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:58:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:58:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:58:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:58:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:58:02.550 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:58:02.550 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:58:02.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T22:58:02.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:58:02.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T22:58:02.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T22:58:02.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 32 = '' 2024-03-11T22:58:02.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:58:02.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T22:58:03.353 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:58:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:58:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:58:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:58:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:58:03.407 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:58:03.407 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:58:03.408 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T22:58:03.408 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:58:03.408 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T22:58:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T22:58:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 32 = '' 2024-03-11T22:58:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:58:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T22:58:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:58:05.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:58:05.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:05.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:05.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:58:05.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:58:05.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:58:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:58:05.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=32 2024-03-11T22:58:05.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 32 = 32 2024-03-11T22:58:05.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:58:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2024-03-11T22:58:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:58:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap10 2024-03-11T22:58:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T22:58:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T22:58:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T22:58:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2024-03-11T22:58:06.103 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'test': 2024-03-11T22:58:06.103 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 512 MiB in 128 objects 2024-03-11T22:58:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T22:58:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 20 2024-03-11T22:58:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 10ab50368aa4 2024-03-11T22:58:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.10ab50368aa4 2024-03-11T22:58:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T22:58:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T22:58:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T22:58:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T22:58:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 22:49:42 2024 2024-03-11T22:58:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 22:56:22 2024 2024-03-11T22:58:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 22:49:42 2024 2024-03-11T22:58:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T22:58:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T22:58:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T22:58:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: d6055756-c87a-420f-9f23-02c2bbaa6857 2024-03-11T22:58:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T22:58:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:58:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snap9-clean ']' 2024-03-11T22:58:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap9-clean 2024-03-11T22:58:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap9-clean 2024-03-11T22:58:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:58:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:06.734 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:07.342 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:07.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:07.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:07.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:58:07.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap10 2024-03-11T22:58:07.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:07.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:07.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap10 2024-03-11T22:58:07.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:58:07.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:07.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:07.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:08.250 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:08.259 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:09.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:09.377 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:58:09.377 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ clean_snap_name=snap10-clean 2024-03-11T22:58:09.377 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror test snap10-clean 2024-03-11T22:58:09.377 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:58:09.377 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:09.377 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:09.377 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap10-clean 2024-03-11T22:58:09.377 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2024-03-11T22:58:10.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-11T22:58:10.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:58:10.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:10.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:10.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-11T22:58:10.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:58:10.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T22:58:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T22:58:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:58:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T22:58:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-11T22:58:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:58:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T22:58:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T22:58:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T22:58:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-11T22:58:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T22:58:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T22:58:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:58:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T22:58:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T22:58:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T22:58:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T22:58:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T22:58:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T22:58:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T22:58:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:58:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-11T22:58:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T22:58:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:58:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:58:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T22:58:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T22:58:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:58:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:58:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:58:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T22:58:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T22:58:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:58:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:58:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T22:58:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:58:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T22:58:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T22:58:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:58:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:58:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T22:58:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T22:58:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:58:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:58:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:58:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T22:58:11.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T22:58:11.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:58:11.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:58:11.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T22:58:11.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T22:58:11.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T22:58:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:58:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:58:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T22:58:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-11T22:58:11.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-11T22:58:11.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T22:58:11.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T22:58:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:58:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-03-11T22:58:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T22:58:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:58:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:58:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T22:58:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T22:58:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:58:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:58:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:58:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T22:58:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T22:58:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:58:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:58:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T22:58:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T22:58:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T22:58:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T22:58:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:58:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:58:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T22:58:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T22:58:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:58:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:58:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:58:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T22:58:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T22:58:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:58:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:58:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T22:58:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T22:58:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T22:58:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T22:58:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T22:58:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T22:58:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-03-11T22:58:11.422 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-11T22:58:11.422 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T22:58:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T22:58:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:58:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-03-11T22:58:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T22:58:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:58:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:58:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T22:58:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T22:58:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:58:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:58:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:58:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T22:58:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T22:58:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:58:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:58:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T22:58:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T22:58:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T22:58:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T22:58:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:58:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:58:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T22:58:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T22:58:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:58:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:58:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:58:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T22:58:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T22:58:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:58:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:58:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T22:58:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T22:58:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T22:58:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T22:58:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T22:58:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T22:58:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-03-11T22:58:11.517 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-11T22:58:11.517 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T22:58:11.517 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T22:58:11.520 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T22:58:11.521 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-03-11T22:58:11.521 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:3 2024-03-11T22:58:11.521 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T22:58:11.521 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T22:58:11.521 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-03-11T22:58:11.521 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:3 2024-03-11T22:58:11.521 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T22:58:11.521 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T22:58:11.521 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T22:58:11.521 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=3 2024-03-11T22:58:11.521 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 3 = cluster1:3 ']' 2024-03-11T22:58:11.521 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T22:58:11.521 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:58:11.521 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=3 2024-03-11T22:58:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T22:58:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-03-11T22:58:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:3 2024-03-11T22:58:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T22:58:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T22:58:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-03-11T22:58:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:3 2024-03-11T22:58:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T22:58:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T22:58:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T22:58:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T22:58:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 3 = cluster1:3 ']' 2024-03-11T22:58:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T22:58:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T22:58:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=3 2024-03-11T22:58:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=3 2024-03-11T22:58:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-03-11T22:58:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T22:58:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T22:58:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T22:58:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-03-11T22:58:11.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T22:58:11.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T22:58:11.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T22:58:11.620 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T22:58:11.620 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T22:58:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T22:58:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T22:58:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:58:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-11T22:58:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T22:58:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:58:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T22:58:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T22:58:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-03-11T22:58:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T22:58:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:58:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-test.status 2024-03-11T22:58:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-test.status 2024-03-11T22:58:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T22:58:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.status 2024-03-11T22:58:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-test.status 2024-03-11T22:58:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-test.status 2024-03-11T22:58:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T22:58:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:58:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-03-11T22:58:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:58:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-03-11T22:58:12.779 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 34 2024-03-11T22:58:12.789 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.status 2024-03-11T22:58:12.789 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:58:12.789 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:12.789 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:12.789 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.status 2024-03-11T22:58:12.790 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-03-11T22:58:12.790 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:58:12.872 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T22:58:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=34 2024-03-11T22:58:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:58:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:58:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T22:58:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:58:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:58:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:58:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:58:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:58:13.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:58:13.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:58:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T22:58:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:58:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T22:58:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T22:58:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 34 = '' 2024-03-11T22:58:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:58:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T22:58:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:58:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:58:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:58:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:58:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:58:13.578 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T22:58:13.578 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:58:13.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T22:58:13.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T22:58:13.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T22:58:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T22:58:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 34 = '' 2024-03-11T22:58:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T22:58:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T22:58:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:58:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:58:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.status 2024-03-11T22:58:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-03-11T22:58:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T22:58:14.432 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T22:58:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=34 2024-03-11T22:58:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 34 = 34 2024-03-11T22:58:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:58:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2024-03-11T22:58:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:58:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap10-clean 2024-03-11T22:58:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T22:58:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T22:58:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T22:58:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2024-03-11T22:58:15.485 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'test': 2024-03-11T22:58:15.485 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 512 MiB in 128 objects 2024-03-11T22:58:15.485 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T22:58:15.485 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 21 2024-03-11T22:58:15.485 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 10ab50368aa4 2024-03-11T22:58:15.485 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.10ab50368aa4 2024-03-11T22:58:15.485 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T22:58:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T22:58:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T22:58:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T22:58:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 22:49:42 2024 2024-03-11T22:58:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 22:58:06 2024 2024-03-11T22:58:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 22:49:42 2024 2024-03-11T22:58:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T22:58:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T22:58:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T22:58:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: d6055756-c87a-420f-9f23-02c2bbaa6857 2024-03-11T22:58:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T22:58:15.489 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:58:15.489 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ seq 1 10 2024-03-11T22:58:15.490 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T22:58:15.490 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap1 2024-03-11T22:58:15.491 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap1 2024-03-11T22:58:15.491 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:15.491 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:15.491 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap1 2024-03-11T22:58:15.491 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:58:15.491 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:15.491 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:15.491 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:15.491 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:17.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:58:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap1-clean 2024-03-11T22:58:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap1-clean 2024-03-11T22:58:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap1-clean 2024-03-11T22:58:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:58:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:17.819 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:18.562 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:18.573 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:58:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T22:58:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap2 2024-03-11T22:58:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap2 2024-03-11T22:58:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap2 2024-03-11T22:58:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:58:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:19.507 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:20.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:20.543 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:58:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap2-clean 2024-03-11T22:58:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap2-clean 2024-03-11T22:58:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap2-clean 2024-03-11T22:58:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:58:20.661 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:20.661 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:20.661 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:20.661 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:21.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:21.258 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:22.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:22.239 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:58:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T22:58:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap3 2024-03-11T22:58:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap3 2024-03-11T22:58:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap3 2024-03-11T22:58:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:58:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:23.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:23.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:23.909 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:58:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap3-clean 2024-03-11T22:58:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap3-clean 2024-03-11T22:58:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap3-clean 2024-03-11T22:58:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:58:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:24.642 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:24.652 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:25.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:58:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T22:58:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap4 2024-03-11T22:58:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap4 2024-03-11T22:58:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap4 2024-03-11T22:58:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:58:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:27.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:58:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap4-clean 2024-03-11T22:58:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap4-clean 2024-03-11T22:58:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap4-clean 2024-03-11T22:58:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:58:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:28.097 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:28.906 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:29.116 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:58:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T22:58:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap5 2024-03-11T22:58:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap5 2024-03-11T22:58:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap5 2024-03-11T22:58:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:58:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:29.828 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:29.839 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:30.646 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:58:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap5-clean 2024-03-11T22:58:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap5-clean 2024-03-11T22:58:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap5-clean 2024-03-11T22:58:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:58:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:31.590 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:32.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:32.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:32.587 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:58:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T22:58:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap6 2024-03-11T22:58:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap6 2024-03-11T22:58:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap6 2024-03-11T22:58:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:58:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:34.208 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:58:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap6-clean 2024-03-11T22:58:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap6-clean 2024-03-11T22:58:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap6-clean 2024-03-11T22:58:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:58:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:34.920 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:35.678 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:35.690 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:35.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:58:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T22:58:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap7 2024-03-11T22:58:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap7 2024-03-11T22:58:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap7 2024-03-11T22:58:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:58:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:36.673 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6482291899620419) > weight (0.25) 2024-03-11T22:58:36.673 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.3136082820537939) > weight (0.25) 2024-03-11T22:58:36.673 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-03-11T22:58:36.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-03-11T22:58:36.673 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-03-11T22:58:36.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi132.stderr:2024-03-11T22:58:36.670+0000 7f0c451ff640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 42128) UID: 1000 2024-03-11T22:58:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:37.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:58:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap7-clean 2024-03-11T22:58:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap7-clean 2024-03-11T22:58:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap7-clean 2024-03-11T22:58:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:58:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:38.509 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:38.521 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:39.352 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:39.362 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:39.687 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:58:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T22:58:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap8 2024-03-11T22:58:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap8 2024-03-11T22:58:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap8 2024-03-11T22:58:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:58:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:40.311 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:41.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:58:41.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap8-clean 2024-03-11T22:58:41.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap8-clean 2024-03-11T22:58:41.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:41.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:41.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap8-clean 2024-03-11T22:58:41.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:58:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:41.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:42.933 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:58:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T22:58:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap9 2024-03-11T22:58:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap9 2024-03-11T22:58:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap9 2024-03-11T22:58:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:58:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:43.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:43.674 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-03-11T22:58:43.674 INFO:teuthology.orchestra.run:waiting for 600 2024-03-11T22:58:43.674 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-03-11T22:58:43.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-03-11T22:58:43.675 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-03-11T22:58:43.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-03-11T22:58:43.678 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2024-03-11T22:58:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:44.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:44.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:58:44.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap9-clean 2024-03-11T22:58:44.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap9-clean 2024-03-11T22:58:44.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:44.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:44.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap9-clean 2024-03-11T22:58:44.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:58:44.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:44.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:44.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:44.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:45.447 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:46.572 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:46.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:58:46.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T22:58:46.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap10 2024-03-11T22:58:46.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap10 2024-03-11T22:58:46.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:46.689 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:46.689 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap10 2024-03-11T22:58:46.689 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:58:46.689 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:46.689 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:46.689 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:46.689 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:47.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:47.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:48.176 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:58:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap10-clean 2024-03-11T22:58:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap10-clean 2024-03-11T22:58:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap10-clean 2024-03-11T22:58:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T22:58:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export 2024-03-11T22:58:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-03-11T22:58:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:49.889 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-test.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-test.export 2024-03-11T22:58:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:58:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ seq 1 10 2024-03-11T22:58:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T22:58:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap1 2024-03-11T22:58:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror test snap1 2024-03-11T22:58:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:58:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap1 2024-03-11T22:58:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2024-03-11T22:58:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T22:58:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap1-clean 2024-03-11T22:58:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2024-03-11T22:58:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:58:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap1-clean 2024-03-11T22:58:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2024-03-11T22:58:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T22:58:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T22:58:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap2 2024-03-11T22:58:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror test snap2 2024-03-11T22:58:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:58:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:51.789 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap2 2024-03-11T22:58:51.789 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2024-03-11T22:58:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T22:58:52.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap2-clean 2024-03-11T22:58:52.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2024-03-11T22:58:52.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:58:52.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:52.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:52.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap2-clean 2024-03-11T22:58:52.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2024-03-11T22:58:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T22:58:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T22:58:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap3 2024-03-11T22:58:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror test snap3 2024-03-11T22:58:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:58:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap3 2024-03-11T22:58:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2024-03-11T22:58:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T22:58:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap3-clean 2024-03-11T22:58:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2024-03-11T22:58:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:58:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap3-clean 2024-03-11T22:58:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2024-03-11T22:58:55.787 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T22:58:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T22:58:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap4 2024-03-11T22:58:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror test snap4 2024-03-11T22:58:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:58:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap4 2024-03-11T22:58:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2024-03-11T22:58:56.785 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T22:58:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap4-clean 2024-03-11T22:58:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2024-03-11T22:58:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:58:56.795 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:56.795 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:56.795 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap4-clean 2024-03-11T22:58:56.795 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2024-03-11T22:58:57.780 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-03-11T22:58:57.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-03-11T22:58:57.781 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-03-11T22:58:57.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr:2024-03-11T22:58:57.777+0000 7efe71c31640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 45435) UID: 1000 2024-03-11T22:58:57.797 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T22:58:57.811 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T22:58:57.812 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap5 2024-03-11T22:58:57.812 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror test snap5 2024-03-11T22:58:57.812 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:58:57.812 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:57.812 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:57.812 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap5 2024-03-11T22:58:57.812 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2024-03-11T22:58:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T22:58:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap5-clean 2024-03-11T22:58:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2024-03-11T22:58:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:58:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap5-clean 2024-03-11T22:58:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2024-03-11T22:58:59.224 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T22:58:59.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T22:58:59.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap6 2024-03-11T22:58:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror test snap6 2024-03-11T22:58:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:58:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:58:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:58:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap6 2024-03-11T22:58:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2024-03-11T22:59:00.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T22:59:00.250 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap6-clean 2024-03-11T22:59:00.250 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2024-03-11T22:59:00.250 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:59:00.250 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:00.250 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:59:00.250 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap6-clean 2024-03-11T22:59:00.250 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2024-03-11T22:59:01.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T22:59:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T22:59:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap7 2024-03-11T22:59:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror test snap7 2024-03-11T22:59:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:59:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:59:01.256 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap7 2024-03-11T22:59:01.256 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2024-03-11T22:59:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T22:59:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap7-clean 2024-03-11T22:59:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2024-03-11T22:59:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:59:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:59:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap7-clean 2024-03-11T22:59:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2024-03-11T22:59:03.249 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T22:59:03.260 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T22:59:03.260 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap8 2024-03-11T22:59:03.260 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror test snap8 2024-03-11T22:59:03.260 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:59:03.260 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:03.260 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:59:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap8 2024-03-11T22:59:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2024-03-11T22:59:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T22:59:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap8-clean 2024-03-11T22:59:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2024-03-11T22:59:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:59:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:59:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap8-clean 2024-03-11T22:59:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2024-03-11T22:59:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T22:59:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T22:59:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap9 2024-03-11T22:59:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror test snap9 2024-03-11T22:59:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:59:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:59:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap9 2024-03-11T22:59:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2024-03-11T22:59:06.265 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T22:59:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap9-clean 2024-03-11T22:59:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2024-03-11T22:59:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:59:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:59:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap9-clean 2024-03-11T22:59:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2024-03-11T22:59:06.781 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-03-11T22:59:06.781 INFO:teuthology.orchestra.run:waiting for 600 2024-03-11T22:59:06.782 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-03-11T22:59:06.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-03-11T22:59:06.782 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-03-11T22:59:06.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-03-11T22:59:06.785 INFO:tasks.rbd_mirror_thrash:waiting for 119 secs before thrashing 2024-03-11T22:59:07.270 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T22:59:07.286 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T22:59:07.286 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap10 2024-03-11T22:59:07.286 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror test snap10 2024-03-11T22:59:07.287 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:59:07.287 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:07.287 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:59:07.287 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap10 2024-03-11T22:59:07.287 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2024-03-11T22:59:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T22:59:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap10-clean 2024-03-11T22:59:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2024-03-11T22:59:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:59:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:59:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap10-clean 2024-03-11T22:59:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2024-03-11T22:59:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T22:59:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror test 2024-03-11T22:59:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:59:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:59:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T22:59:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T22:59:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror test 2024-03-11T22:59:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:59:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:09.243 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:59:09.243 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2024-03-11T22:59:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/test 2024-03-11T22:59:09.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi132.stderr:2024-03-11T22:59:09.378+0000 7fc1b3009640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558190e756c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T22:59:09.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi132.stderr:2024-03-11T22:59:09.378+0000 7fc1ba017640 -1 rbd::mirror::ImageReplayer: 0x55819179a280 [3/d6055756-c87a-420f-9f23-02c2bbaa6857] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T22:59:09.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi132.stderr:2024-03-11T22:59:09.733+0000 7fc1b3009640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558190e75860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T22:59:09.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi132.stderr:2024-03-11T22:59:09.733+0000 7fc1ba017640 -1 rbd::mirror::ImageReplayer: 0x55819179a280 [3/d6055756-c87a-420f-9f23-02c2bbaa6857] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T22:59:10.495 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-03-11T22:59:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:59:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror test deleted 2024-03-11T22:59:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:59:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:59:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T22:59:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T22:59:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T22:59:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T22:59:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror test 2024-03-11T22:59:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T22:59:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T22:59:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=test 2024-03-11T22:59:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-03-11T22:59:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T22:59:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image test: (2) No such file or directory 2024-03-11T22:59:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T22:59:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T22:59:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T22:59:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror test deleted '' 2024-03-11T22:59:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T22:59:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=test 2024-03-11T22:59:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T22:59:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T22:59:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T22:59:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T22:59:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror test 2024-03-11T22:59:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T22:59:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T22:59:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=test 2024-03-11T22:59:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-03-11T22:59:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T22:59:10.663 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image test: (2) No such file or directory 2024-03-11T22:59:10.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T22:59:10.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T22:59:10.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T22:59:10.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:59:10.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ testlog 'TEST: create many images' 2024-03-11T22:59:10.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ tee -a /tmp/tmp.7mIVT0l9FS/rbd-mirror.test.log 2024-03-11T22:59:10.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ date '+%F %T' 2024-03-11T22:59:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo 2024-03-11 22:59:10 TEST: create many images 2024-03-11T22:59:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:2024-03-11 22:59:10 TEST: create many images 2024-03-11T22:59:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ snap_name=snap 2024-03-11T22:59:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ seq 1 50 2024-03-11T22:59:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T22:59:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_1 2024-03-11T22:59:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 snapshot 128M 2024-03-11T22:59:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:59:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T22:59:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T22:59:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T22:59:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_1 128M 2024-03-11T22:59:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:59:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T22:59:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T22:59:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T22:59:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T22:59:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_1 2024-03-11T22:59:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T22:59:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T22:59:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:59:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_1 snapshot 2024-03-11T22:59:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:59:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T22:59:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T22:59:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2024-03-11T22:59:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T22:59:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-03-11T22:59:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_1': 2024-03-11T22:59:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T22:59:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T22:59:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T22:59:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 125e7152d7d6 2024-03-11T22:59:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.125e7152d7d6 2024-03-11T22:59:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T22:59:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T22:59:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T22:59:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T22:59:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 22:59:10 2024 2024-03-11T22:59:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 22:59:10 2024 2024-03-11T22:59:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 22:59:10 2024 2024-03-11T22:59:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T22:59:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T22:59:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 7380f9c0-c386-40d3-8842-2968a362a7ba 2024-03-11T22:59:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T22:59:11.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T22:59:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_1 2024-03-11T22:59:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:59:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T22:59:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T22:59:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=10 2024-03-11T22:59:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T22:59:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-03-11T22:59:11.518 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 45350912 bytes 2024-03-11T22:59:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 103396352 bytes 2024-03-11T22:59:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 146457600 bytes 2024-03-11T22:59:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 272731136 bytes 2024-03-11T22:59:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 335016960 bytes 2024-03-11T22:59:12.608 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 365701120 bytes 2024-03-11T22:59:12.732 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 394602496 bytes 2024-03-11T22:59:12.876 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 409607168 bytes 2024-03-11T22:59:13.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 462571520 bytes 2024-03-11T22:59:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 512715776 bytes 2024-03-11T22:59:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 567457792 bytes 2024-03-11T22:59:13.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 605911040 bytes 2024-03-11T22:59:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 624266240 bytes 2024-03-11T22:59:13.910 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 678098944 bytes 2024-03-11T22:59:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 724162560 bytes 2024-03-11T22:59:14.230 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 742208512 bytes 2024-03-11T22:59:14.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 775001088 bytes 2024-03-11T22:59:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 820285440 bytes 2024-03-11T22:59:14.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 852904960 bytes 2024-03-11T22:59:14.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 897952768 bytes 2024-03-11T22:59:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 937740288 bytes 2024-03-11T22:59:15.169 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 954976256 bytes 2024-03-11T22:59:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 1005115392 bytes 2024-03-11T22:59:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 1080870912 bytes 2024-03-11T22:59:15.747 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1131255808 bytes 2024-03-11T22:59:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1141457920 bytes 2024-03-11T22:59:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1198404608 bytes 2024-03-11T22:59:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1242989568 bytes 2024-03-11T22:59:16.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1281950720 bytes 2024-03-11T22:59:16.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1307566080 bytes 2024-03-11T22:59:16.613 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1326497792 bytes 2024-03-11T22:59:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1350497280 bytes 2024-03-11T22:59:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1382901760 bytes 2024-03-11T22:59:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1415887872 bytes 2024-03-11T22:59:17.178 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1455188992 bytes 2024-03-11T22:59:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1476866048 bytes 2024-03-11T22:59:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1519258624 bytes 2024-03-11T22:59:17.608 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1540617216 bytes 2024-03-11T22:59:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1558195200 bytes 2024-03-11T22:59:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1629298688 bytes 2024-03-11T22:59:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1685160960 bytes 2024-03-11T22:59:18.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1765243904 bytes 2024-03-11T22:59:18.551 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1812999168 bytes 2024-03-11T22:59:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 1872392192 bytes 2024-03-11T22:59:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 1926969344 bytes 2024-03-11T22:59:19.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 2016415744 bytes 2024-03-11T22:59:19.432 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 2072700928 bytes 2024-03-11T22:59:19.616 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 2131057664 bytes 2024-03-11T22:59:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 2131678208 bytes 2024-03-11T22:59:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 2153193472 bytes 2024-03-11T22:59:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5100 IOs, 2174327808 bytes 2024-03-11T22:59:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5200 IOs, 2210905088 bytes 2024-03-11T22:59:20.287 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5300 IOs, 2239465472 bytes 2024-03-11T22:59:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5400 IOs, 2262599680 bytes 2024-03-11T22:59:20.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5500 IOs, 2326608896 bytes 2024-03-11T22:59:20.809 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5600 IOs, 2367848448 bytes 2024-03-11T22:59:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5700 IOs, 2388934656 bytes 2024-03-11T22:59:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5800 IOs, 2414185472 bytes 2024-03-11T22:59:21.228 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5900 IOs, 2486321152 bytes 2024-03-11T22:59:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T22:59:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T22:59:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T22:59:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:59:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T22:59:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_2 2024-03-11T22:59:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 snapshot 128M 2024-03-11T22:59:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:59:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T22:59:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T22:59:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T22:59:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_2 128M 2024-03-11T22:59:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:59:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T22:59:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T22:59:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T22:59:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T22:59:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_2 2024-03-11T22:59:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T22:59:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T22:59:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:59:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_2 snapshot 2024-03-11T22:59:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:59:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T22:59:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T22:59:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2024-03-11T22:59:22.251 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T22:59:22.256 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-03-11T22:59:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_2': 2024-03-11T22:59:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T22:59:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T22:59:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T22:59:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 126842474392 2024-03-11T22:59:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.126842474392 2024-03-11T22:59:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T22:59:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T22:59:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T22:59:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T22:59:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 22:59:21 2024 2024-03-11T22:59:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 22:59:21 2024 2024-03-11T22:59:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 22:59:21 2024 2024-03-11T22:59:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T22:59:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T22:59:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 0db5ab54-08c1-4a9b-b8f1-d2ac90c55bee 2024-03-11T22:59:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T22:59:22.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T22:59:22.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_2 2024-03-11T22:59:22.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:59:22.298 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:22.298 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T22:59:22.298 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T22:59:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=12 2024-03-11T22:59:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T22:59:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-03-11T22:59:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 15286272 bytes 2024-03-11T22:59:22.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 50612224 bytes 2024-03-11T22:59:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 51218432 bytes 2024-03-11T22:59:22.825 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 104039424 bytes 2024-03-11T22:59:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 145838080 bytes 2024-03-11T22:59:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 160560128 bytes 2024-03-11T22:59:23.211 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 196172800 bytes 2024-03-11T22:59:23.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 213166080 bytes 2024-03-11T22:59:23.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 347822080 bytes 2024-03-11T22:59:23.794 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 362744832 bytes 2024-03-11T22:59:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 424119296 bytes 2024-03-11T22:59:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 459476992 bytes 2024-03-11T22:59:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 538175488 bytes 2024-03-11T22:59:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 607283200 bytes 2024-03-11T22:59:24.796 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 647653376 bytes 2024-03-11T22:59:24.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 695517184 bytes 2024-03-11T22:59:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 712701952 bytes 2024-03-11T22:59:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 795621376 bytes 2024-03-11T22:59:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 856037376 bytes 2024-03-11T22:59:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 858714112 bytes 2024-03-11T22:59:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 877848576 bytes 2024-03-11T22:59:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 976221184 bytes 2024-03-11T22:59:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 1016792064 bytes 2024-03-11T22:59:26.666 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 1130701824 bytes 2024-03-11T22:59:26.818 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1161186304 bytes 2024-03-11T22:59:26.922 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1175113728 bytes 2024-03-11T22:59:27.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1235397632 bytes 2024-03-11T22:59:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1318933504 bytes 2024-03-11T22:59:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1319529472 bytes 2024-03-11T22:59:27.572 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1339233280 bytes 2024-03-11T22:59:27.841 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1435403264 bytes 2024-03-11T22:59:27.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1462093824 bytes 2024-03-11T22:59:28.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1487420416 bytes 2024-03-11T22:59:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1589914624 bytes 2024-03-11T22:59:28.522 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1623322624 bytes 2024-03-11T22:59:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1716992000 bytes 2024-03-11T22:59:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1732840448 bytes 2024-03-11T22:59:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1842672640 bytes 2024-03-11T22:59:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1864886272 bytes 2024-03-11T22:59:29.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1936207872 bytes 2024-03-11T22:59:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1999963136 bytes 2024-03-11T22:59:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 2022880256 bytes 2024-03-11T22:59:30.059 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 2061736960 bytes 2024-03-11T22:59:30.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 2107141120 bytes 2024-03-11T22:59:30.432 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 2178836480 bytes 2024-03-11T22:59:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 2180984832 bytes 2024-03-11T22:59:30.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 2211510272 bytes 2024-03-11T22:59:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 2287608832 bytes 2024-03-11T22:59:31.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 2371923968 bytes 2024-03-11T22:59:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 2401638400 bytes 2024-03-11T22:59:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5100 IOs, 2458277888 bytes 2024-03-11T22:59:31.706 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5200 IOs, 2525827072 bytes 2024-03-11T22:59:31.862 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5300 IOs, 2576293888 bytes 2024-03-11T22:59:32.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5400 IOs, 2629702656 bytes 2024-03-11T22:59:32.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5500 IOs, 2669263872 bytes 2024-03-11T22:59:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5600 IOs, 2683916288 bytes 2024-03-11T22:59:32.537 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5700 IOs, 2740667392 bytes 2024-03-11T22:59:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5800 IOs, 2785073152 bytes 2024-03-11T22:59:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5900 IOs, 2809090048 bytes 2024-03-11T22:59:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6000 IOs, 2855547904 bytes 2024-03-11T22:59:33.135 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6100 IOs, 2894640128 bytes 2024-03-11T22:59:33.244 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6200 IOs, 2901980160 bytes 2024-03-11T22:59:33.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6300 IOs, 2929503232 bytes 2024-03-11T22:59:33.470 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6400 IOs, 2957103104 bytes 2024-03-11T22:59:33.731 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6500 IOs, 3034514432 bytes 2024-03-11T22:59:33.866 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6600 IOs, 3072879616 bytes 2024-03-11T22:59:33.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6700 IOs, 3073497088 bytes 2024-03-11T22:59:34.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6800 IOs, 3127596032 bytes 2024-03-11T22:59:34.294 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6900 IOs, 3151663104 bytes 2024-03-11T22:59:34.304 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T22:59:34.304 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T22:59:34.304 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T22:59:34.304 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:59:34.304 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T22:59:34.304 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_3 2024-03-11T22:59:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 snapshot 128M 2024-03-11T22:59:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:59:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T22:59:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T22:59:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T22:59:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_3 128M 2024-03-11T22:59:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:59:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T22:59:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T22:59:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T22:59:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T22:59:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_3 2024-03-11T22:59:34.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T22:59:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T22:59:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:59:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_3 snapshot 2024-03-11T22:59:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:59:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T22:59:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T22:59:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2024-03-11T22:59:34.838 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T22:59:34.842 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-03-11T22:59:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_3': 2024-03-11T22:59:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T22:59:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T22:59:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T22:59:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 127079d53de3 2024-03-11T22:59:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.127079d53de3 2024-03-11T22:59:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T22:59:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T22:59:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T22:59:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T22:59:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 22:59:34 2024 2024-03-11T22:59:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 22:59:34 2024 2024-03-11T22:59:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 22:59:34 2024 2024-03-11T22:59:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T22:59:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T22:59:34.880 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 89903746-8b7b-4dea-b492-96c6f9aebf71 2024-03-11T22:59:34.880 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T22:59:34.882 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T22:59:34.882 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_3 2024-03-11T22:59:34.882 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:59:34.882 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:34.882 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T22:59:34.882 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T22:59:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=11 2024-03-11T22:59:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T22:59:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-03-11T22:59:35.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 9912320 bytes 2024-03-11T22:59:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 37740544 bytes 2024-03-11T22:59:35.321 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 81347584 bytes 2024-03-11T22:59:35.479 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 123293696 bytes 2024-03-11T22:59:35.635 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 154919936 bytes 2024-03-11T22:59:35.782 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 194781184 bytes 2024-03-11T22:59:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 238268416 bytes 2024-03-11T22:59:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 279481344 bytes 2024-03-11T22:59:36.242 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 298808320 bytes 2024-03-11T22:59:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 364255232 bytes 2024-03-11T22:59:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 409540608 bytes 2024-03-11T22:59:36.796 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 459635712 bytes 2024-03-11T22:59:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 460273664 bytes 2024-03-11T22:59:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 491933696 bytes 2024-03-11T22:59:37.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 504214528 bytes 2024-03-11T22:59:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 519606272 bytes 2024-03-11T22:59:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 617061376 bytes 2024-03-11T22:59:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 619611136 bytes 2024-03-11T22:59:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 667738112 bytes 2024-03-11T22:59:37.896 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 691178496 bytes 2024-03-11T22:59:38.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 710464512 bytes 2024-03-11T22:59:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 758140928 bytes 2024-03-11T22:59:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 793978880 bytes 2024-03-11T22:59:38.616 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 859813888 bytes 2024-03-11T22:59:38.848 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 949645312 bytes 2024-03-11T22:59:39.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1014024192 bytes 2024-03-11T22:59:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1138278400 bytes 2024-03-11T22:59:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1142323200 bytes 2024-03-11T22:59:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1146963968 bytes 2024-03-11T22:59:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1150673920 bytes 2024-03-11T22:59:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1215050752 bytes 2024-03-11T22:59:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1266992128 bytes 2024-03-11T22:59:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1354719232 bytes 2024-03-11T22:59:40.563 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1423544320 bytes 2024-03-11T22:59:40.738 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1471043584 bytes 2024-03-11T22:59:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1510262784 bytes 2024-03-11T22:59:41.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1545403392 bytes 2024-03-11T22:59:41.221 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1576188928 bytes 2024-03-11T22:59:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1655982080 bytes 2024-03-11T22:59:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1695054848 bytes 2024-03-11T22:59:41.728 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1709283328 bytes 2024-03-11T22:59:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1740358656 bytes 2024-03-11T22:59:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1759302656 bytes 2024-03-11T22:59:42.159 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 1811544064 bytes 2024-03-11T22:59:42.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 1860321280 bytes 2024-03-11T22:59:42.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 1884947456 bytes 2024-03-11T22:59:42.622 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 1909114880 bytes 2024-03-11T22:59:42.778 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 1933523968 bytes 2024-03-11T22:59:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 1979303936 bytes 2024-03-11T22:59:43.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 2003352576 bytes 2024-03-11T22:59:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5100 IOs, 2009149440 bytes 2024-03-11T22:59:43.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5200 IOs, 2058110976 bytes 2024-03-11T22:59:43.561 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5300 IOs, 2131633152 bytes 2024-03-11T22:59:43.686 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5400 IOs, 2159788032 bytes 2024-03-11T22:59:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5500 IOs, 2204247040 bytes 2024-03-11T22:59:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5600 IOs, 2251038720 bytes 2024-03-11T22:59:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5700 IOs, 2285923328 bytes 2024-03-11T22:59:44.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5800 IOs, 2325207040 bytes 2024-03-11T22:59:44.525 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5900 IOs, 2366728192 bytes 2024-03-11T22:59:44.702 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6000 IOs, 2440395776 bytes 2024-03-11T22:59:44.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6100 IOs, 2474051584 bytes 2024-03-11T22:59:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6200 IOs, 2536951808 bytes 2024-03-11T22:59:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6300 IOs, 2562849792 bytes 2024-03-11T22:59:45.303 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6400 IOs, 2563445760 bytes 2024-03-11T22:59:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6500 IOs, 2584004608 bytes 2024-03-11T22:59:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6600 IOs, 2660641792 bytes 2024-03-11T22:59:45.810 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6700 IOs, 2686768128 bytes 2024-03-11T22:59:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T22:59:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T22:59:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T22:59:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:59:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T22:59:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_4 2024-03-11T22:59:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 snapshot 128M 2024-03-11T22:59:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:59:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T22:59:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T22:59:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T22:59:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_4 128M 2024-03-11T22:59:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:59:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T22:59:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T22:59:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T22:59:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T22:59:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_4 2024-03-11T22:59:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T22:59:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T22:59:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:59:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_4 snapshot 2024-03-11T22:59:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:59:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T22:59:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T22:59:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2024-03-11T22:59:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T22:59:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-03-11T22:59:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_4': 2024-03-11T22:59:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T22:59:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T22:59:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T22:59:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 12799e33458d 2024-03-11T22:59:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.12799e33458d 2024-03-11T22:59:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T22:59:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T22:59:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T22:59:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T22:59:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 22:59:45 2024 2024-03-11T22:59:46.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 22:59:45 2024 2024-03-11T22:59:46.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 22:59:45 2024 2024-03-11T22:59:46.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T22:59:46.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T22:59:46.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: fc8576a3-dbba-4c16-95dd-f75d6193b45b 2024-03-11T22:59:46.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T22:59:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T22:59:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_4 2024-03-11T22:59:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:59:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T22:59:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T22:59:46.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=10 2024-03-11T22:59:46.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T22:59:46.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-03-11T22:59:46.529 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 60599296 bytes 2024-03-11T22:59:46.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 145605632 bytes 2024-03-11T22:59:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 185697280 bytes 2024-03-11T22:59:47.047 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 206311424 bytes 2024-03-11T22:59:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 244013056 bytes 2024-03-11T22:59:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 274864128 bytes 2024-03-11T22:59:47.738 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 339490816 bytes 2024-03-11T22:59:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 395332608 bytes 2024-03-11T22:59:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 400404480 bytes 2024-03-11T22:59:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 406078464 bytes 2024-03-11T22:59:48.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 469189632 bytes 2024-03-11T22:59:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 523975680 bytes 2024-03-11T22:59:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 541380608 bytes 2024-03-11T22:59:48.845 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 594531328 bytes 2024-03-11T22:59:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 651927552 bytes 2024-03-11T22:59:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 729076736 bytes 2024-03-11T22:59:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 758175744 bytes 2024-03-11T22:59:49.560 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 788395008 bytes 2024-03-11T22:59:49.743 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 840725504 bytes 2024-03-11T22:59:49.945 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 895738880 bytes 2024-03-11T22:59:50.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 952646656 bytes 2024-03-11T22:59:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 1023121408 bytes 2024-03-11T22:59:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 1062265856 bytes 2024-03-11T22:59:50.739 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 1139628032 bytes 2024-03-11T22:59:50.827 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1140243456 bytes 2024-03-11T22:59:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1194848256 bytes 2024-03-11T22:59:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1278083072 bytes 2024-03-11T22:59:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1309529088 bytes 2024-03-11T22:59:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1339471872 bytes 2024-03-11T22:59:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1447955456 bytes 2024-03-11T22:59:52.086 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1530514432 bytes 2024-03-11T22:59:52.258 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1553491968 bytes 2024-03-11T22:59:52.383 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1574781952 bytes 2024-03-11T22:59:52.561 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1619843072 bytes 2024-03-11T22:59:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1689439232 bytes 2024-03-11T22:59:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1744252928 bytes 2024-03-11T22:59:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1758228480 bytes 2024-03-11T22:59:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1818886144 bytes 2024-03-11T22:59:53.514 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1877195776 bytes 2024-03-11T22:59:53.597 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1895492608 bytes 2024-03-11T22:59:53.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1941285888 bytes 2024-03-11T22:59:53.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1962318848 bytes 2024-03-11T22:59:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1984091136 bytes 2024-03-11T22:59:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 2010985472 bytes 2024-03-11T22:59:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 2055322624 bytes 2024-03-11T22:59:54.585 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 2107585536 bytes 2024-03-11T22:59:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 2128111616 bytes 2024-03-11T22:59:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 2164572160 bytes 2024-03-11T22:59:55.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 2189236224 bytes 2024-03-11T22:59:55.228 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 2244442112 bytes 2024-03-11T22:59:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5100 IOs, 2353473536 bytes 2024-03-11T22:59:55.766 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5200 IOs, 2427619328 bytes 2024-03-11T22:59:56.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5300 IOs, 2507511808 bytes 2024-03-11T22:59:56.113 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5400 IOs, 2516276224 bytes 2024-03-11T22:59:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5500 IOs, 2538321920 bytes 2024-03-11T22:59:56.302 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T22:59:56.302 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T22:59:56.302 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T22:59:56.302 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:59:56.302 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T22:59:56.302 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_5 2024-03-11T22:59:56.302 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 snapshot 128M 2024-03-11T22:59:56.302 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:59:56.303 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:56.303 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:56.303 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:56.303 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T22:59:56.303 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:56.303 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T22:59:56.303 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T22:59:56.303 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:56.303 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_5 128M 2024-03-11T22:59:56.303 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:59:56.303 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:56.303 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:56.303 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T22:59:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T22:59:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T22:59:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T22:59:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T22:59:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_5 2024-03-11T22:59:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T22:59:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T22:59:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T22:59:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_5 snapshot 2024-03-11T22:59:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:59:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T22:59:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T22:59:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2024-03-11T22:59:56.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T22:59:56.786 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-03-11T22:59:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_5': 2024-03-11T22:59:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T22:59:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T22:59:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T22:59:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 12838003789e 2024-03-11T22:59:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.12838003789e 2024-03-11T22:59:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T22:59:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T22:59:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T22:59:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T22:59:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 22:59:56 2024 2024-03-11T22:59:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 22:59:56 2024 2024-03-11T22:59:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 22:59:56 2024 2024-03-11T22:59:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T22:59:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T22:59:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 6e7c5b9b-f490-4f68-8ae3-de136df2e981 2024-03-11T22:59:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T22:59:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T22:59:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_5 2024-03-11T22:59:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T22:59:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T22:59:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T22:59:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T22:59:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=12 2024-03-11T22:59:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T22:59:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-03-11T22:59:57.085 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 70471680 bytes 2024-03-11T22:59:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 148780032 bytes 2024-03-11T22:59:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 165614592 bytes 2024-03-11T22:59:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 180516864 bytes 2024-03-11T22:59:57.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 256441344 bytes 2024-03-11T22:59:57.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 301111296 bytes 2024-03-11T22:59:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 317991936 bytes 2024-03-11T22:59:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 343493632 bytes 2024-03-11T22:59:58.331 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 368716800 bytes 2024-03-11T22:59:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 387525632 bytes 2024-03-11T22:59:58.588 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 417227776 bytes 2024-03-11T22:59:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 465025024 bytes 2024-03-11T22:59:58.872 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 477737984 bytes 2024-03-11T22:59:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 478337024 bytes 2024-03-11T22:59:59.084 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 511380480 bytes 2024-03-11T22:59:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 533685248 bytes 2024-03-11T22:59:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 664889344 bytes 2024-03-11T22:59:59.766 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 724038656 bytes 2024-03-11T22:59:59.903 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 757814272 bytes 2024-03-11T23:00:00.014 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 771439616 bytes 2024-03-11T23:00:00.120 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 784747520 bytes 2024-03-11T23:00:00.258 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 810625024 bytes 2024-03-11T23:00:00.415 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 845845504 bytes 2024-03-11T23:00:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 899342336 bytes 2024-03-11T23:00:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 947391488 bytes 2024-03-11T23:00:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 965533696 bytes 2024-03-11T23:00:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 996162560 bytes 2024-03-11T23:00:01.209 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1031064576 bytes 2024-03-11T23:00:01.309 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1063638016 bytes 2024-03-11T23:00:01.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1115589632 bytes 2024-03-11T23:00:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1135375360 bytes 2024-03-11T23:00:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1154440192 bytes 2024-03-11T23:00:02.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1242119168 bytes 2024-03-11T23:00:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1260524544 bytes 2024-03-11T23:00:02.287 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1294836736 bytes 2024-03-11T23:00:02.415 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1326208000 bytes 2024-03-11T23:00:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1347452928 bytes 2024-03-11T23:00:02.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1359318016 bytes 2024-03-11T23:00:02.868 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1415052288 bytes 2024-03-11T23:00:02.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1416916992 bytes 2024-03-11T23:00:03.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1514745856 bytes 2024-03-11T23:00:03.421 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1561264128 bytes 2024-03-11T23:00:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1577595904 bytes 2024-03-11T23:00:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 1673525248 bytes 2024-03-11T23:00:04.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 1718381568 bytes 2024-03-11T23:00:04.181 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 1775601664 bytes 2024-03-11T23:00:04.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 1796908032 bytes 2024-03-11T23:00:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 1862268928 bytes 2024-03-11T23:00:04.752 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 1965235200 bytes 2024-03-11T23:00:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 2032795648 bytes 2024-03-11T23:00:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5100 IOs, 2095621120 bytes 2024-03-11T23:00:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5200 IOs, 2120033280 bytes 2024-03-11T23:00:05.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5300 IOs, 2164459520 bytes 2024-03-11T23:00:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5400 IOs, 2204600320 bytes 2024-03-11T23:00:05.851 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5500 IOs, 2252753920 bytes 2024-03-11T23:00:05.988 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5600 IOs, 2253365248 bytes 2024-03-11T23:00:06.154 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5700 IOs, 2310420480 bytes 2024-03-11T23:00:06.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5800 IOs, 2338565120 bytes 2024-03-11T23:00:06.531 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5900 IOs, 2393809920 bytes 2024-03-11T23:00:06.679 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-03-11T23:00:06.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-03-11T23:00:06.679 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-03-11T23:00:06.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi132.stderr:2024-03-11T23:00:06.676+0000 7f847db09640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 44773) UID: 1000 2024-03-11T23:00:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6000 IOs, 2433180672 bytes 2024-03-11T23:00:06.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6100 IOs, 2525490176 bytes 2024-03-11T23:00:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6200 IOs, 2543309824 bytes 2024-03-11T23:00:07.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6300 IOs, 2632393728 bytes 2024-03-11T23:00:07.483 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6400 IOs, 2644472832 bytes 2024-03-11T23:00:07.694 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6500 IOs, 2708516864 bytes 2024-03-11T23:00:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6600 IOs, 2718460928 bytes 2024-03-11T23:00:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6700 IOs, 2773171200 bytes 2024-03-11T23:00:08.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6800 IOs, 2805638144 bytes 2024-03-11T23:00:08.265 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6900 IOs, 2833047552 bytes 2024-03-11T23:00:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7000 IOs, 2920442880 bytes 2024-03-11T23:00:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7100 IOs, 2988965888 bytes 2024-03-11T23:00:08.834 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:00:08.834 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:00:08.834 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:00:08.834 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:00:08.834 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:00:08.834 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_6 2024-03-11T23:00:08.834 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 snapshot 128M 2024-03-11T23:00:08.834 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:00:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:00:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:00:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:00:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:00:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_6 128M 2024-03-11T23:00:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:00:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:00:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:00:08.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:08.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:00:08.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:00:08.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:00:08.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:08.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_6 2024-03-11T23:00:08.880 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:00:08.880 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:00:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:00:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:00:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_6 snapshot 2024-03-11T23:00:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:00:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:00:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:00:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:00:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2024-03-11T23:00:09.905 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:00:09.911 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-03-11T23:00:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_6': 2024-03-11T23:00:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:00:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:00:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:00:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 128cb988c663 2024-03-11T23:00:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.128cb988c663 2024-03-11T23:00:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:00:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:00:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:00:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:00:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:00:08 2024 2024-03-11T23:00:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:00:08 2024 2024-03-11T23:00:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:00:08 2024 2024-03-11T23:00:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:00:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:00:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: fdce66ca-291f-45dc-b094-a088660b842f 2024-03-11T23:00:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:00:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:00:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_6 2024-03-11T23:00:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:00:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:00:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:00:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:00:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=9 2024-03-11T23:00:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:00:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:00:10.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 24561664 bytes 2024-03-11T23:00:10.249 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 32673792 bytes 2024-03-11T23:00:10.412 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 75738112 bytes 2024-03-11T23:00:10.682 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 166017024 bytes 2024-03-11T23:00:10.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 216031232 bytes 2024-03-11T23:00:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 223071232 bytes 2024-03-11T23:00:11.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 253178880 bytes 2024-03-11T23:00:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 277508096 bytes 2024-03-11T23:00:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 313158656 bytes 2024-03-11T23:00:11.518 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 339493888 bytes 2024-03-11T23:00:11.680 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-03-11T23:00:11.680 INFO:teuthology.orchestra.run:waiting for 600 2024-03-11T23:00:11.680 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-03-11T23:00:11.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-03-11T23:00:11.681 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-03-11T23:00:11.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-03-11T23:00:11.684 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2024-03-11T23:00:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 385144832 bytes 2024-03-11T23:00:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 427073536 bytes 2024-03-11T23:00:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 474210304 bytes 2024-03-11T23:00:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 547870720 bytes 2024-03-11T23:00:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 600970240 bytes 2024-03-11T23:00:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 678053888 bytes 2024-03-11T23:00:12.843 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 703399936 bytes 2024-03-11T23:00:12.962 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 714361856 bytes 2024-03-11T23:00:13.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 751325184 bytes 2024-03-11T23:00:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 836575232 bytes 2024-03-11T23:00:13.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 893071360 bytes 2024-03-11T23:00:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 959654912 bytes 2024-03-11T23:00:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 970579968 bytes 2024-03-11T23:00:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 1012382720 bytes 2024-03-11T23:00:14.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1013391360 bytes 2024-03-11T23:00:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1018215424 bytes 2024-03-11T23:00:14.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1074214912 bytes 2024-03-11T23:00:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1151580160 bytes 2024-03-11T23:00:14.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1242132480 bytes 2024-03-11T23:00:15.080 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1257181184 bytes 2024-03-11T23:00:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1324111872 bytes 2024-03-11T23:00:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1393573888 bytes 2024-03-11T23:00:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1437989888 bytes 2024-03-11T23:00:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1438604288 bytes 2024-03-11T23:00:15.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1446721536 bytes 2024-03-11T23:00:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1481377792 bytes 2024-03-11T23:00:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1507286016 bytes 2024-03-11T23:00:16.533 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1548779520 bytes 2024-03-11T23:00:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1651271680 bytes 2024-03-11T23:00:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1694501888 bytes 2024-03-11T23:00:17.176 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1742855168 bytes 2024-03-11T23:00:17.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1744059392 bytes 2024-03-11T23:00:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1770171392 bytes 2024-03-11T23:00:17.544 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 1791075328 bytes 2024-03-11T23:00:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 1849428992 bytes 2024-03-11T23:00:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 1862445056 bytes 2024-03-11T23:00:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 1886807040 bytes 2024-03-11T23:00:18.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 1933075456 bytes 2024-03-11T23:00:18.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 1984192512 bytes 2024-03-11T23:00:18.542 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 2048803840 bytes 2024-03-11T23:00:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5100 IOs, 2170303488 bytes 2024-03-11T23:00:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:00:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:00:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:00:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:00:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:00:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_7 2024-03-11T23:00:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 snapshot 128M 2024-03-11T23:00:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:00:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:00:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:00:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:00:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:00:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_7 128M 2024-03-11T23:00:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:00:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:00:18.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:18.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:00:18.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:18.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:00:18.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:00:18.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:00:18.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:18.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_7 2024-03-11T23:00:19.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:00:19.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:00:19.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:00:19.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:00:19.006 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_7 snapshot 2024-03-11T23:00:19.006 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:00:19.006 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:00:19.006 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:00:19.006 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:00:19.006 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2024-03-11T23:00:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:00:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-03-11T23:00:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_7': 2024-03-11T23:00:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:00:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:00:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:00:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 129bb0e466c0 2024-03-11T23:00:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.129bb0e466c0 2024-03-11T23:00:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:00:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:00:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:00:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:00:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:00:18 2024 2024-03-11T23:00:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:00:18 2024 2024-03-11T23:00:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:00:18 2024 2024-03-11T23:00:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:00:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:00:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: e7d22bec-3c5d-4227-9c8e-5fb052d5be91 2024-03-11T23:00:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:00:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:00:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_7 2024-03-11T23:00:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:00:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:00:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:00:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:00:19.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=13 2024-03-11T23:00:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:00:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:00:20.129 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 6846464 bytes 2024-03-11T23:00:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 102212608 bytes 2024-03-11T23:00:20.572 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 146263040 bytes 2024-03-11T23:00:20.800 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 218949632 bytes 2024-03-11T23:00:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 224215040 bytes 2024-03-11T23:00:21.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 235357184 bytes 2024-03-11T23:00:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 269007872 bytes 2024-03-11T23:00:21.359 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 341430272 bytes 2024-03-11T23:00:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 363154432 bytes 2024-03-11T23:00:21.608 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 400114688 bytes 2024-03-11T23:00:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 421715968 bytes 2024-03-11T23:00:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 460318720 bytes 2024-03-11T23:00:22.095 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 503018496 bytes 2024-03-11T23:00:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 517039104 bytes 2024-03-11T23:00:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 538390528 bytes 2024-03-11T23:00:22.519 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 585277440 bytes 2024-03-11T23:00:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 622004224 bytes 2024-03-11T23:00:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 655496192 bytes 2024-03-11T23:00:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 694801408 bytes 2024-03-11T23:00:23.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 739517440 bytes 2024-03-11T23:00:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 759585792 bytes 2024-03-11T23:00:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 823546880 bytes 2024-03-11T23:00:23.695 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 884609024 bytes 2024-03-11T23:00:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 917284864 bytes 2024-03-11T23:00:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 982543360 bytes 2024-03-11T23:00:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1007562752 bytes 2024-03-11T23:00:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1017454592 bytes 2024-03-11T23:00:24.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1114307584 bytes 2024-03-11T23:00:24.728 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1144879104 bytes 2024-03-11T23:00:24.847 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1158694912 bytes 2024-03-11T23:00:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1215052800 bytes 2024-03-11T23:00:25.180 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1237259264 bytes 2024-03-11T23:00:25.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1281142784 bytes 2024-03-11T23:00:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1302027264 bytes 2024-03-11T23:00:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1367090176 bytes 2024-03-11T23:00:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1376503808 bytes 2024-03-11T23:00:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1417650176 bytes 2024-03-11T23:00:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1473641472 bytes 2024-03-11T23:00:26.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1511987200 bytes 2024-03-11T23:00:26.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1553933312 bytes 2024-03-11T23:00:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1587141632 bytes 2024-03-11T23:00:26.695 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1587763200 bytes 2024-03-11T23:00:26.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1640743936 bytes 2024-03-11T23:00:27.082 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 1709993984 bytes 2024-03-11T23:00:27.221 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 1712908288 bytes 2024-03-11T23:00:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 1716527104 bytes 2024-03-11T23:00:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 1764617216 bytes 2024-03-11T23:00:27.651 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 1810016256 bytes 2024-03-11T23:00:27.823 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 1852285952 bytes 2024-03-11T23:00:28.014 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 1905062912 bytes 2024-03-11T23:00:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5100 IOs, 1942000640 bytes 2024-03-11T23:00:28.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5200 IOs, 2001214464 bytes 2024-03-11T23:00:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5300 IOs, 2074277888 bytes 2024-03-11T23:00:28.766 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5400 IOs, 2103444480 bytes 2024-03-11T23:00:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5500 IOs, 2109239296 bytes 2024-03-11T23:00:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5600 IOs, 2149799936 bytes 2024-03-11T23:00:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5700 IOs, 2217233408 bytes 2024-03-11T23:00:29.490 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5800 IOs, 2281824256 bytes 2024-03-11T23:00:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5900 IOs, 2294669312 bytes 2024-03-11T23:00:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6000 IOs, 2339122176 bytes 2024-03-11T23:00:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6100 IOs, 2344634368 bytes 2024-03-11T23:00:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6200 IOs, 2390594560 bytes 2024-03-11T23:00:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6300 IOs, 2412418048 bytes 2024-03-11T23:00:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6400 IOs, 2416968704 bytes 2024-03-11T23:00:30.410 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6500 IOs, 2456373248 bytes 2024-03-11T23:00:30.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6600 IOs, 2485118976 bytes 2024-03-11T23:00:30.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6700 IOs, 2524646400 bytes 2024-03-11T23:00:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6800 IOs, 2584886272 bytes 2024-03-11T23:00:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6900 IOs, 2623187968 bytes 2024-03-11T23:00:31.255 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7000 IOs, 2662881280 bytes 2024-03-11T23:00:31.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7100 IOs, 2694918144 bytes 2024-03-11T23:00:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7200 IOs, 2695540736 bytes 2024-03-11T23:00:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7300 IOs, 2724415488 bytes 2024-03-11T23:00:31.854 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7400 IOs, 2792189952 bytes 2024-03-11T23:00:32.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7500 IOs, 2850357248 bytes 2024-03-11T23:00:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7600 IOs, 2887764992 bytes 2024-03-11T23:00:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7700 IOs, 2928890880 bytes 2024-03-11T23:00:32.590 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7800 IOs, 2990356480 bytes 2024-03-11T23:00:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7900 IOs, 3052954624 bytes 2024-03-11T23:00:32.883 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:8000 IOs, 3053548544 bytes 2024-03-11T23:00:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:00:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:00:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:00:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:00:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:00:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_8 2024-03-11T23:00:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 snapshot 128M 2024-03-11T23:00:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:00:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:33.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:00:33.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:33.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_8 2024-03-11T23:00:33.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:33.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:00:33.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:00:33.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:33.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_8 128M 2024-03-11T23:00:33.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:00:33.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:33.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:00:33.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:33.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_8 2024-03-11T23:00:33.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:33.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:00:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:00:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:00:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_8 2024-03-11T23:00:33.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:00:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:00:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:00:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:00:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_8 snapshot 2024-03-11T23:00:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:00:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:00:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_8 2024-03-11T23:00:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:00:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2024-03-11T23:00:34.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:00:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-03-11T23:00:34.066 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_8': 2024-03-11T23:00:34.066 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:00:34.066 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:00:34.066 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:00:34.066 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 12a3daa63864 2024-03-11T23:00:34.066 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.12a3daa63864 2024-03-11T23:00:34.066 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:00:34.066 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:00:34.066 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:00:34.066 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:00:34.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:00:33 2024 2024-03-11T23:00:34.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:00:33 2024 2024-03-11T23:00:34.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:00:33 2024 2024-03-11T23:00:34.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:00:34.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:00:34.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: ac93902e-f852-49af-ac1f-b13ea2b5269b 2024-03-11T23:00:34.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:00:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:00:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_8 2024-03-11T23:00:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:00:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:00:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_8 2024-03-11T23:00:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:00:34.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=9 2024-03-11T23:00:34.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:00:34.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:00:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 39868416 bytes 2024-03-11T23:00:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 65600512 bytes 2024-03-11T23:00:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 80801792 bytes 2024-03-11T23:00:34.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 105596928 bytes 2024-03-11T23:00:34.752 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 138242048 bytes 2024-03-11T23:00:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 198850560 bytes 2024-03-11T23:00:35.199 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 270987264 bytes 2024-03-11T23:00:35.429 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 339622912 bytes 2024-03-11T23:00:35.572 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 367742976 bytes 2024-03-11T23:00:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 438865920 bytes 2024-03-11T23:00:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 472858624 bytes 2024-03-11T23:00:36.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 473699328 bytes 2024-03-11T23:00:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 508703744 bytes 2024-03-11T23:00:36.316 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 531831808 bytes 2024-03-11T23:00:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 598311936 bytes 2024-03-11T23:00:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 685640704 bytes 2024-03-11T23:00:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 723313664 bytes 2024-03-11T23:00:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 752969728 bytes 2024-03-11T23:00:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 871721984 bytes 2024-03-11T23:00:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 886963200 bytes 2024-03-11T23:00:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 918837248 bytes 2024-03-11T23:00:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 961829888 bytes 2024-03-11T23:00:38.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 1002321920 bytes 2024-03-11T23:00:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 1002944512 bytes 2024-03-11T23:00:38.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1084787712 bytes 2024-03-11T23:00:38.537 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1138458624 bytes 2024-03-11T23:00:38.723 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1186724864 bytes 2024-03-11T23:00:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1202369536 bytes 2024-03-11T23:00:38.988 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1228513280 bytes 2024-03-11T23:00:39.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1244694528 bytes 2024-03-11T23:00:39.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1293575168 bytes 2024-03-11T23:00:39.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1304342528 bytes 2024-03-11T23:00:39.521 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1333045248 bytes 2024-03-11T23:00:39.687 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1370438656 bytes 2024-03-11T23:00:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1385539584 bytes 2024-03-11T23:00:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1415332864 bytes 2024-03-11T23:00:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1453213696 bytes 2024-03-11T23:00:40.335 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1554981888 bytes 2024-03-11T23:00:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1566214144 bytes 2024-03-11T23:00:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1579320320 bytes 2024-03-11T23:00:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1594963968 bytes 2024-03-11T23:00:40.845 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1615697920 bytes 2024-03-11T23:00:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1652212736 bytes 2024-03-11T23:00:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 1697072128 bytes 2024-03-11T23:00:41.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 1745806336 bytes 2024-03-11T23:00:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 1762039808 bytes 2024-03-11T23:00:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 1762642944 bytes 2024-03-11T23:00:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 1794971648 bytes 2024-03-11T23:00:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 1825243136 bytes 2024-03-11T23:00:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 1894848512 bytes 2024-03-11T23:00:42.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5100 IOs, 1920222208 bytes 2024-03-11T23:00:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5200 IOs, 1956486144 bytes 2024-03-11T23:00:42.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5300 IOs, 1957087232 bytes 2024-03-11T23:00:42.837 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5400 IOs, 1987174400 bytes 2024-03-11T23:00:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5500 IOs, 2009440256 bytes 2024-03-11T23:00:43.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:00:43.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:00:43.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:00:43.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:00:43.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:00:43.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_9 2024-03-11T23:00:43.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 snapshot 128M 2024-03-11T23:00:43.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:00:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:00:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_9 2024-03-11T23:00:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:00:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:00:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_9 128M 2024-03-11T23:00:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:00:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:00:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_9 2024-03-11T23:00:43.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:43.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:00:43.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:00:43.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:00:43.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:43.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_9 2024-03-11T23:00:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:00:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:00:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:00:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:00:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_9 snapshot 2024-03-11T23:00:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:00:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:00:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_9 2024-03-11T23:00:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:00:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2024-03-11T23:00:44.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:00:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-03-11T23:00:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_9': 2024-03-11T23:00:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:00:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:00:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:00:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 12acae174655 2024-03-11T23:00:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.12acae174655 2024-03-11T23:00:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:00:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:00:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:00:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:00:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:00:43 2024 2024-03-11T23:00:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:00:43 2024 2024-03-11T23:00:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:00:43 2024 2024-03-11T23:00:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:00:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:00:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: b02ac2fb-bf75-42d9-bf3f-a8404d4fe797 2024-03-11T23:00:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:00:44.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:00:44.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_9 2024-03-11T23:00:44.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:00:44.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:00:44.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_9 2024-03-11T23:00:44.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:00:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=14 2024-03-11T23:00:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:00:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:00:44.410 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 46189568 bytes 2024-03-11T23:00:44.545 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 67805184 bytes 2024-03-11T23:00:44.699 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 98439168 bytes 2024-03-11T23:00:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 150739968 bytes 2024-03-11T23:00:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 207667200 bytes 2024-03-11T23:00:45.223 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 238734336 bytes 2024-03-11T23:00:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 287441920 bytes 2024-03-11T23:00:45.560 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 337854464 bytes 2024-03-11T23:00:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 406678528 bytes 2024-03-11T23:00:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 409417728 bytes 2024-03-11T23:00:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 469366784 bytes 2024-03-11T23:00:46.274 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 488036352 bytes 2024-03-11T23:00:46.383 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 501995520 bytes 2024-03-11T23:00:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 547070976 bytes 2024-03-11T23:00:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 628868096 bytes 2024-03-11T23:00:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 673011712 bytes 2024-03-11T23:00:47.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 714748928 bytes 2024-03-11T23:00:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 823195648 bytes 2024-03-11T23:00:47.559 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 843710464 bytes 2024-03-11T23:00:47.741 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 888683520 bytes 2024-03-11T23:00:47.888 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 921447424 bytes 2024-03-11T23:00:48.008 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 942686208 bytes 2024-03-11T23:00:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 961399808 bytes 2024-03-11T23:00:48.270 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 990614528 bytes 2024-03-11T23:00:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1071850496 bytes 2024-03-11T23:00:48.649 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1092789248 bytes 2024-03-11T23:00:48.788 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1117742080 bytes 2024-03-11T23:00:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1159230464 bytes 2024-03-11T23:00:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1220192256 bytes 2024-03-11T23:00:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1232224256 bytes 2024-03-11T23:00:49.492 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1322070016 bytes 2024-03-11T23:00:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1336830976 bytes 2024-03-11T23:00:49.776 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1357597696 bytes 2024-03-11T23:00:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1417992192 bytes 2024-03-11T23:00:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1511839744 bytes 2024-03-11T23:00:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1552925696 bytes 2024-03-11T23:00:50.524 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1556985856 bytes 2024-03-11T23:00:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1617265664 bytes 2024-03-11T23:00:50.908 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1660141568 bytes 2024-03-11T23:00:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1660770304 bytes 2024-03-11T23:00:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1710451712 bytes 2024-03-11T23:00:51.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1752222720 bytes 2024-03-11T23:00:51.488 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1788001280 bytes 2024-03-11T23:00:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 1801561088 bytes 2024-03-11T23:00:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 1875318784 bytes 2024-03-11T23:00:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 1882262528 bytes 2024-03-11T23:00:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 2025989120 bytes 2024-03-11T23:00:52.407 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 2039348224 bytes 2024-03-11T23:00:52.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 2111485952 bytes 2024-03-11T23:00:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 2248206336 bytes 2024-03-11T23:00:53.152 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5100 IOs, 2278987776 bytes 2024-03-11T23:00:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5200 IOs, 2327627776 bytes 2024-03-11T23:00:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5300 IOs, 2385517568 bytes 2024-03-11T23:00:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5400 IOs, 2450450432 bytes 2024-03-11T23:00:53.870 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5500 IOs, 2469668864 bytes 2024-03-11T23:00:54.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5600 IOs, 2525257728 bytes 2024-03-11T23:00:54.288 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5700 IOs, 2576370688 bytes 2024-03-11T23:00:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5800 IOs, 2692700160 bytes 2024-03-11T23:00:54.800 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5900 IOs, 2740590592 bytes 2024-03-11T23:00:55.072 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6000 IOs, 2834636800 bytes 2024-03-11T23:00:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6100 IOs, 2900265984 bytes 2024-03-11T23:00:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6200 IOs, 2922211328 bytes 2024-03-11T23:00:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6300 IOs, 3023106048 bytes 2024-03-11T23:00:55.899 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6400 IOs, 3070554112 bytes 2024-03-11T23:00:56.178 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6500 IOs, 3153469440 bytes 2024-03-11T23:00:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6600 IOs, 3197487104 bytes 2024-03-11T23:00:56.486 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6700 IOs, 3245898752 bytes 2024-03-11T23:00:56.742 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6800 IOs, 3301924864 bytes 2024-03-11T23:00:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6900 IOs, 3352930304 bytes 2024-03-11T23:00:57.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7000 IOs, 3391618048 bytes 2024-03-11T23:00:57.223 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7100 IOs, 3417421824 bytes 2024-03-11T23:00:57.322 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7200 IOs, 3425888256 bytes 2024-03-11T23:00:57.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7300 IOs, 3428824064 bytes 2024-03-11T23:00:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7400 IOs, 3485085696 bytes 2024-03-11T23:00:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7500 IOs, 3508474880 bytes 2024-03-11T23:00:57.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7600 IOs, 3558288384 bytes 2024-03-11T23:00:58.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:00:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:00:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:00:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:00:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:00:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_10 2024-03-11T23:00:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 snapshot 128M 2024-03-11T23:00:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:00:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:00:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_10 2024-03-11T23:00:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:00:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:00:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_10 128M 2024-03-11T23:00:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:00:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:00:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_10 2024-03-11T23:00:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:00:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:00:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:00:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:00:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_10 2024-03-11T23:00:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:00:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:00:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:00:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:00:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_10 snapshot 2024-03-11T23:00:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:00:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:00:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_10 2024-03-11T23:00:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:00:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2024-03-11T23:00:59.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:00:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-03-11T23:00:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_10': 2024-03-11T23:00:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:00:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:00:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:00:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 12b5fd3b5e65 2024-03-11T23:00:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.12b5fd3b5e65 2024-03-11T23:00:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:00:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:00:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:00:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:00:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:00:58 2024 2024-03-11T23:00:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:00:58 2024 2024-03-11T23:00:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:00:58 2024 2024-03-11T23:00:59.270 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:00:59.270 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:00:59.270 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 6e7e7d8e-c0d0-4968-90fa-7731ffa367f1 2024-03-11T23:00:59.270 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:00:59.272 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:00:59.272 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_10 2024-03-11T23:00:59.272 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:00:59.273 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:00:59.273 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_10 2024-03-11T23:00:59.273 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:00:59.274 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=6 2024-03-11T23:00:59.274 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:00:59.275 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:00:59.444 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 14445568 bytes 2024-03-11T23:00:59.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 94161920 bytes 2024-03-11T23:00:59.810 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 111593472 bytes 2024-03-11T23:01:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 165598208 bytes 2024-03-11T23:01:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 170576896 bytes 2024-03-11T23:01:00.321 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 221405184 bytes 2024-03-11T23:01:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 263936000 bytes 2024-03-11T23:01:00.722 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 312443904 bytes 2024-03-11T23:01:00.932 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 364400640 bytes 2024-03-11T23:01:01.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 404943872 bytes 2024-03-11T23:01:01.309 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 481686528 bytes 2024-03-11T23:01:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 528209920 bytes 2024-03-11T23:01:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 606515200 bytes 2024-03-11T23:01:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 641838080 bytes 2024-03-11T23:01:02.047 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 659877888 bytes 2024-03-11T23:01:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 660483072 bytes 2024-03-11T23:01:02.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 707751936 bytes 2024-03-11T23:01:02.424 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 708362240 bytes 2024-03-11T23:01:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 737492992 bytes 2024-03-11T23:01:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 811416576 bytes 2024-03-11T23:01:02.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 840757248 bytes 2024-03-11T23:01:03.170 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 897297408 bytes 2024-03-11T23:01:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 964169728 bytes 2024-03-11T23:01:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 1029620736 bytes 2024-03-11T23:01:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1052957696 bytes 2024-03-11T23:01:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1111656448 bytes 2024-03-11T23:01:04.085 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1128986624 bytes 2024-03-11T23:01:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1195238400 bytes 2024-03-11T23:01:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1224832000 bytes 2024-03-11T23:01:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1242217472 bytes 2024-03-11T23:01:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1305340928 bytes 2024-03-11T23:01:04.884 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1311089664 bytes 2024-03-11T23:01:05.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1327671296 bytes 2024-03-11T23:01:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1391875072 bytes 2024-03-11T23:01:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:01:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:01:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:01:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:01:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:01:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_11 2024-03-11T23:01:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 snapshot 128M 2024-03-11T23:01:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:01:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:01:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_11 2024-03-11T23:01:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:01:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:01:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_11 128M 2024-03-11T23:01:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:01:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:01:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_11 2024-03-11T23:01:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:01:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:01:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:01:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_11 2024-03-11T23:01:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:01:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:01:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:01:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:01:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_11 snapshot 2024-03-11T23:01:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:01:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:01:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_11 2024-03-11T23:01:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:01:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_11 snapshot 2024-03-11T23:01:05.786 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7655735037820763) > weight (0.25) 2024-03-11T23:01:05.786 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7623101841491031) > weight (0.25) 2024-03-11T23:01:05.786 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.8903809495365879) > weight (0.25) 2024-03-11T23:01:05.786 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.420517124073453) > weight (0.25) 2024-03-11T23:01:05.786 INFO:tasks.rbd_mirror_thrash:waiting for 98 secs before thrashing 2024-03-11T23:01:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:01:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_11 2024-03-11T23:01:06.296 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_11': 2024-03-11T23:01:06.296 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:01:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:01:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:01:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 12bd15a5dd53 2024-03-11T23:01:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.12bd15a5dd53 2024-03-11T23:01:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:01:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:01:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:01:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:01:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:01:05 2024 2024-03-11T23:01:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:01:05 2024 2024-03-11T23:01:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:01:05 2024 2024-03-11T23:01:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:01:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:01:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 952b2674-c455-44d6-b523-5d182beac4cc 2024-03-11T23:01:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:01:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:01:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_11 2024-03-11T23:01:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:01:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:01:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_11 2024-03-11T23:01:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:01:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=8 2024-03-11T23:01:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:01:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:01:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 54812672 bytes 2024-03-11T23:01:06.665 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 78369792 bytes 2024-03-11T23:01:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 146672640 bytes 2024-03-11T23:01:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 159536128 bytes 2024-03-11T23:01:07.154 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 204599296 bytes 2024-03-11T23:01:07.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 267326464 bytes 2024-03-11T23:01:07.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 321098752 bytes 2024-03-11T23:01:07.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 374630400 bytes 2024-03-11T23:01:07.855 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 375755776 bytes 2024-03-11T23:01:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 402710528 bytes 2024-03-11T23:01:08.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 441871360 bytes 2024-03-11T23:01:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 482547712 bytes 2024-03-11T23:01:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 495788032 bytes 2024-03-11T23:01:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 518569984 bytes 2024-03-11T23:01:08.703 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 566702080 bytes 2024-03-11T23:01:08.839 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 595642368 bytes 2024-03-11T23:01:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 638025728 bytes 2024-03-11T23:01:09.211 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 702381056 bytes 2024-03-11T23:01:09.432 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 769644544 bytes 2024-03-11T23:01:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 855636992 bytes 2024-03-11T23:01:09.926 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 924818432 bytes 2024-03-11T23:01:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 963375104 bytes 2024-03-11T23:01:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 1007393792 bytes 2024-03-11T23:01:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 1019805696 bytes 2024-03-11T23:01:10.556 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1054457856 bytes 2024-03-11T23:01:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1077278720 bytes 2024-03-11T23:01:10.960 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1177999360 bytes 2024-03-11T23:01:11.085 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1211271168 bytes 2024-03-11T23:01:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1214622720 bytes 2024-03-11T23:01:11.335 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1241570304 bytes 2024-03-11T23:01:11.438 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1254957056 bytes 2024-03-11T23:01:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1298102272 bytes 2024-03-11T23:01:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1387657216 bytes 2024-03-11T23:01:12.008 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1433186304 bytes 2024-03-11T23:01:12.298 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1522275328 bytes 2024-03-11T23:01:12.477 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1581059072 bytes 2024-03-11T23:01:12.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1600664576 bytes 2024-03-11T23:01:12.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1626039296 bytes 2024-03-11T23:01:12.975 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1684898816 bytes 2024-03-11T23:01:13.200 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1751609344 bytes 2024-03-11T23:01:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1803451392 bytes 2024-03-11T23:01:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1828060160 bytes 2024-03-11T23:01:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1853250560 bytes 2024-03-11T23:01:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 1885421568 bytes 2024-03-11T23:01:14.015 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 1942976512 bytes 2024-03-11T23:01:14.184 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 1978849280 bytes 2024-03-11T23:01:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:01:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:01:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:01:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:01:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:01:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_12 2024-03-11T23:01:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 snapshot 128M 2024-03-11T23:01:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:01:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:01:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_12 2024-03-11T23:01:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:01:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:01:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_12 128M 2024-03-11T23:01:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:01:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:01:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_12 2024-03-11T23:01:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:01:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:01:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:01:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_12 2024-03-11T23:01:14.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:01:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:01:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:01:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:01:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_12 snapshot 2024-03-11T23:01:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:01:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:01:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_12 2024-03-11T23:01:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:01:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_12 snapshot 2024-03-11T23:01:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:01:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_12 2024-03-11T23:01:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_12': 2024-03-11T23:01:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:01:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:01:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:01:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 12c7ff92f1a9 2024-03-11T23:01:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.12c7ff92f1a9 2024-03-11T23:01:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:01:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:01:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:01:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:01:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:01:14 2024 2024-03-11T23:01:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:01:14 2024 2024-03-11T23:01:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:01:14 2024 2024-03-11T23:01:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:01:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:01:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: f99801f5-5b32-4c9d-b6d4-607f161fc716 2024-03-11T23:01:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:01:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:01:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_12 2024-03-11T23:01:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:01:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:01:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_12 2024-03-11T23:01:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:01:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=12 2024-03-11T23:01:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:01:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:01:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 77503488 bytes 2024-03-11T23:01:15.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 135772160 bytes 2024-03-11T23:01:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 148944896 bytes 2024-03-11T23:01:15.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 200507392 bytes 2024-03-11T23:01:15.788 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 251906048 bytes 2024-03-11T23:01:15.988 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 282993664 bytes 2024-03-11T23:01:16.086 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 291727360 bytes 2024-03-11T23:01:16.227 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 345067520 bytes 2024-03-11T23:01:16.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 358419456 bytes 2024-03-11T23:01:16.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 408127488 bytes 2024-03-11T23:01:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 428612608 bytes 2024-03-11T23:01:16.847 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 462033920 bytes 2024-03-11T23:01:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 539324416 bytes 2024-03-11T23:01:17.217 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 559895552 bytes 2024-03-11T23:01:17.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 568705024 bytes 2024-03-11T23:01:17.555 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 658453504 bytes 2024-03-11T23:01:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 710290432 bytes 2024-03-11T23:01:17.914 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 730508288 bytes 2024-03-11T23:01:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 764354560 bytes 2024-03-11T23:01:18.251 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 805071872 bytes 2024-03-11T23:01:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 850099200 bytes 2024-03-11T23:01:18.573 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 876640256 bytes 2024-03-11T23:01:18.794 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 941603840 bytes 2024-03-11T23:01:18.988 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 1000502272 bytes 2024-03-11T23:01:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1001988096 bytes 2024-03-11T23:01:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1025381376 bytes 2024-03-11T23:01:19.470 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1112341504 bytes 2024-03-11T23:01:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1136290816 bytes 2024-03-11T23:01:19.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1175775232 bytes 2024-03-11T23:01:20.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1261562880 bytes 2024-03-11T23:01:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1322401792 bytes 2024-03-11T23:01:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1346215936 bytes 2024-03-11T23:01:20.466 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1348106240 bytes 2024-03-11T23:01:20.583 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1366651904 bytes 2024-03-11T23:01:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1414251520 bytes 2024-03-11T23:01:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1503996928 bytes 2024-03-11T23:01:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1532325888 bytes 2024-03-11T23:01:21.361 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1573102592 bytes 2024-03-11T23:01:21.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1653962752 bytes 2024-03-11T23:01:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1677479936 bytes 2024-03-11T23:01:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1724292096 bytes 2024-03-11T23:01:22.283 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1757582336 bytes 2024-03-11T23:01:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1762435072 bytes 2024-03-11T23:01:22.555 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 1812550656 bytes 2024-03-11T23:01:22.680 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 1831710720 bytes 2024-03-11T23:01:22.841 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 1855253504 bytes 2024-03-11T23:01:22.972 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 1887051776 bytes 2024-03-11T23:01:23.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 1909671936 bytes 2024-03-11T23:01:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 1967160320 bytes 2024-03-11T23:01:23.512 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 2021834752 bytes 2024-03-11T23:01:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5100 IOs, 2110284800 bytes 2024-03-11T23:01:23.913 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5200 IOs, 2123666432 bytes 2024-03-11T23:01:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5300 IOs, 2220371968 bytes 2024-03-11T23:01:24.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5400 IOs, 2242727936 bytes 2024-03-11T23:01:24.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5500 IOs, 2263972864 bytes 2024-03-11T23:01:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5600 IOs, 2318531584 bytes 2024-03-11T23:01:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5700 IOs, 2434581504 bytes 2024-03-11T23:01:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5800 IOs, 2457171968 bytes 2024-03-11T23:01:25.238 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5900 IOs, 2465671168 bytes 2024-03-11T23:01:25.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6000 IOs, 2475685888 bytes 2024-03-11T23:01:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6100 IOs, 2501373952 bytes 2024-03-11T23:01:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6200 IOs, 2556662784 bytes 2024-03-11T23:01:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6300 IOs, 2642790400 bytes 2024-03-11T23:01:26.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6400 IOs, 2681565184 bytes 2024-03-11T23:01:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6500 IOs, 2720823296 bytes 2024-03-11T23:01:26.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6600 IOs, 2746613760 bytes 2024-03-11T23:01:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6700 IOs, 2831251456 bytes 2024-03-11T23:01:26.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6800 IOs, 2878731264 bytes 2024-03-11T23:01:26.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:01:26.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:01:26.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:01:26.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:01:26.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:01:26.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_13 2024-03-11T23:01:26.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 snapshot 128M 2024-03-11T23:01:26.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:01:26.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:26.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:01:26.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:26.859 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_13 2024-03-11T23:01:26.859 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:26.859 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:01:26.859 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:01:26.859 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:26.859 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_13 128M 2024-03-11T23:01:26.859 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:01:26.859 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:26.859 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:01:26.859 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:26.859 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_13 2024-03-11T23:01:26.859 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:26.859 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:01:26.859 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:01:26.859 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:01:26.860 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:26.860 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_13 2024-03-11T23:01:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:01:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:01:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:01:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:01:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_13 snapshot 2024-03-11T23:01:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:01:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:01:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_13 2024-03-11T23:01:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:01:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_13 snapshot 2024-03-11T23:01:27.901 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:01:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_13 2024-03-11T23:01:27.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_13': 2024-03-11T23:01:27.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:01:27.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:01:27.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:01:27.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 12d18a944bd5 2024-03-11T23:01:27.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.12d18a944bd5 2024-03-11T23:01:27.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:01:27.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:01:27.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:01:27.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:01:27.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:01:26 2024 2024-03-11T23:01:27.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:01:26 2024 2024-03-11T23:01:27.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:01:26 2024 2024-03-11T23:01:27.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:01:27.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:01:27.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 4e5cb2c2-5e2b-44b7-95cc-f0928b52a2e0 2024-03-11T23:01:27.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:01:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:01:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_13 2024-03-11T23:01:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:01:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:01:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_13 2024-03-11T23:01:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:01:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=11 2024-03-11T23:01:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:01:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:01:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 9697280 bytes 2024-03-11T23:01:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 16096256 bytes 2024-03-11T23:01:28.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 106895360 bytes 2024-03-11T23:01:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 133961728 bytes 2024-03-11T23:01:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 145910784 bytes 2024-03-11T23:01:28.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 210113536 bytes 2024-03-11T23:01:29.092 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 231224320 bytes 2024-03-11T23:01:29.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 252402688 bytes 2024-03-11T23:01:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 257777664 bytes 2024-03-11T23:01:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 303836160 bytes 2024-03-11T23:01:29.678 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 336373760 bytes 2024-03-11T23:01:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 397474816 bytes 2024-03-11T23:01:30.092 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 462265344 bytes 2024-03-11T23:01:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 503179264 bytes 2024-03-11T23:01:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 541511680 bytes 2024-03-11T23:01:30.531 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 564220928 bytes 2024-03-11T23:01:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 605137920 bytes 2024-03-11T23:01:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 607458304 bytes 2024-03-11T23:01:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 621929472 bytes 2024-03-11T23:01:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 708272128 bytes 2024-03-11T23:01:31.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 757270528 bytes 2024-03-11T23:01:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 826143744 bytes 2024-03-11T23:01:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 883083264 bytes 2024-03-11T23:01:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 940217344 bytes 2024-03-11T23:01:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1013179392 bytes 2024-03-11T23:01:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1034122240 bytes 2024-03-11T23:01:32.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1056947200 bytes 2024-03-11T23:01:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1091035136 bytes 2024-03-11T23:01:33.036 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1099728896 bytes 2024-03-11T23:01:33.194 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1146073088 bytes 2024-03-11T23:01:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1207677952 bytes 2024-03-11T23:01:33.572 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1238704128 bytes 2024-03-11T23:01:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1302177792 bytes 2024-03-11T23:01:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1333360640 bytes 2024-03-11T23:01:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1388557312 bytes 2024-03-11T23:01:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1436328960 bytes 2024-03-11T23:01:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1454475264 bytes 2024-03-11T23:01:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1497698304 bytes 2024-03-11T23:01:34.794 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1509793792 bytes 2024-03-11T23:01:35.026 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1587560448 bytes 2024-03-11T23:01:35.130 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1592115200 bytes 2024-03-11T23:01:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1606427648 bytes 2024-03-11T23:01:35.400 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1630671872 bytes 2024-03-11T23:01:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 1639347200 bytes 2024-03-11T23:01:35.670 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 1674686464 bytes 2024-03-11T23:01:35.761 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 1679687680 bytes 2024-03-11T23:01:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 1695864832 bytes 2024-03-11T23:01:36.113 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 1762955264 bytes 2024-03-11T23:01:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 1806852096 bytes 2024-03-11T23:01:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 1881877504 bytes 2024-03-11T23:01:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5100 IOs, 1948005376 bytes 2024-03-11T23:01:37.016 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5200 IOs, 2022718464 bytes 2024-03-11T23:01:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5300 IOs, 2055447552 bytes 2024-03-11T23:01:37.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5400 IOs, 2096193536 bytes 2024-03-11T23:01:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5500 IOs, 2145630208 bytes 2024-03-11T23:01:37.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5600 IOs, 2163296256 bytes 2024-03-11T23:01:37.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5700 IOs, 2230562816 bytes 2024-03-11T23:01:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5800 IOs, 2327890944 bytes 2024-03-11T23:01:38.444 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5900 IOs, 2408666112 bytes 2024-03-11T23:01:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6000 IOs, 2428099584 bytes 2024-03-11T23:01:38.784 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6100 IOs, 2476781568 bytes 2024-03-11T23:01:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6200 IOs, 2486319104 bytes 2024-03-11T23:01:38.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:01:38.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:01:38.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:01:38.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:01:38.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:01:38.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_14 2024-03-11T23:01:38.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 snapshot 128M 2024-03-11T23:01:38.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:01:38.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:38.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:01:38.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_14 2024-03-11T23:01:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:01:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:01:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_14 128M 2024-03-11T23:01:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:01:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:01:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_14 2024-03-11T23:01:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:01:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:01:38.960 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:01:38.960 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:38.960 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_14 2024-03-11T23:01:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:01:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:01:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:01:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:01:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_14 snapshot 2024-03-11T23:01:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:01:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:01:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_14 2024-03-11T23:01:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:01:39.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_14 snapshot 2024-03-11T23:01:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:01:39.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_14 2024-03-11T23:01:39.966 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_14': 2024-03-11T23:01:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:01:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:01:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:01:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 12dbe31781e0 2024-03-11T23:01:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.12dbe31781e0 2024-03-11T23:01:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:01:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:01:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:01:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:01:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:01:38 2024 2024-03-11T23:01:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:01:38 2024 2024-03-11T23:01:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:01:38 2024 2024-03-11T23:01:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:01:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:01:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: e8312f3c-c1b7-4832-92dc-03abff4ac406 2024-03-11T23:01:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:01:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:01:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_14 2024-03-11T23:01:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:01:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:01:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_14 2024-03-11T23:01:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:01:39.972 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=14 2024-03-11T23:01:39.972 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:01:39.972 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:01:40.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 5348352 bytes 2024-03-11T23:01:40.220 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 21396480 bytes 2024-03-11T23:01:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 51776512 bytes 2024-03-11T23:01:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 129359872 bytes 2024-03-11T23:01:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 153551872 bytes 2024-03-11T23:01:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 166998016 bytes 2024-03-11T23:01:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 182918144 bytes 2024-03-11T23:01:41.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 183531520 bytes 2024-03-11T23:01:41.227 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 217604096 bytes 2024-03-11T23:01:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 301776896 bytes 2024-03-11T23:01:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 338578432 bytes 2024-03-11T23:01:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 363990016 bytes 2024-03-11T23:01:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 443734016 bytes 2024-03-11T23:01:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 459378688 bytes 2024-03-11T23:01:42.299 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 482961408 bytes 2024-03-11T23:01:42.457 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 533860352 bytes 2024-03-11T23:01:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 568125440 bytes 2024-03-11T23:01:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 568701952 bytes 2024-03-11T23:01:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 590111744 bytes 2024-03-11T23:01:43.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 621668352 bytes 2024-03-11T23:01:43.253 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 699347968 bytes 2024-03-11T23:01:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 718077952 bytes 2024-03-11T23:01:43.533 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 743890944 bytes 2024-03-11T23:01:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 764684288 bytes 2024-03-11T23:01:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 828235776 bytes 2024-03-11T23:01:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 900287488 bytes 2024-03-11T23:01:44.278 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 961053696 bytes 2024-03-11T23:01:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 996293632 bytes 2024-03-11T23:01:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1036192768 bytes 2024-03-11T23:01:44.768 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1053234176 bytes 2024-03-11T23:01:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1125409792 bytes 2024-03-11T23:01:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1173243904 bytes 2024-03-11T23:01:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1277752320 bytes 2024-03-11T23:01:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1324773376 bytes 2024-03-11T23:01:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1390841856 bytes 2024-03-11T23:01:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1473661952 bytes 2024-03-11T23:01:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1506345984 bytes 2024-03-11T23:01:46.525 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1563423744 bytes 2024-03-11T23:01:46.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1606419456 bytes 2024-03-11T23:01:46.860 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1643927552 bytes 2024-03-11T23:01:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1683945472 bytes 2024-03-11T23:01:47.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1742493696 bytes 2024-03-11T23:01:47.342 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1753758720 bytes 2024-03-11T23:01:47.459 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 1781680128 bytes 2024-03-11T23:01:47.685 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-03-11T23:01:47.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-03-11T23:01:47.686 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-03-11T23:01:47.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi132.stderr:2024-03-11T23:01:47.683+0000 7f72640d7640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 48984) UID: 1000 2024-03-11T23:01:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 1893897216 bytes 2024-03-11T23:01:47.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 1927539712 bytes 2024-03-11T23:01:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 1965708288 bytes 2024-03-11T23:01:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 2007804928 bytes 2024-03-11T23:01:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 2034703360 bytes 2024-03-11T23:01:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 2103197696 bytes 2024-03-11T23:01:48.881 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5100 IOs, 2149847040 bytes 2024-03-11T23:01:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5200 IOs, 2164879360 bytes 2024-03-11T23:01:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5300 IOs, 2186314752 bytes 2024-03-11T23:01:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5400 IOs, 2263048192 bytes 2024-03-11T23:01:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5500 IOs, 2274857984 bytes 2024-03-11T23:01:49.687 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-03-11T23:01:49.687 INFO:teuthology.orchestra.run:waiting for 600 2024-03-11T23:01:49.688 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-03-11T23:01:49.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-03-11T23:01:49.688 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-03-11T23:01:49.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-03-11T23:01:49.691 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2024-03-11T23:01:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5600 IOs, 2360208384 bytes 2024-03-11T23:01:49.862 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5700 IOs, 2368175104 bytes 2024-03-11T23:01:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5800 IOs, 2369740800 bytes 2024-03-11T23:01:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5900 IOs, 2424167424 bytes 2024-03-11T23:01:50.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6000 IOs, 2464273408 bytes 2024-03-11T23:01:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6100 IOs, 2486263808 bytes 2024-03-11T23:01:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6200 IOs, 2519064576 bytes 2024-03-11T23:01:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6300 IOs, 2581607424 bytes 2024-03-11T23:01:51.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6400 IOs, 2659903488 bytes 2024-03-11T23:01:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6500 IOs, 2688075776 bytes 2024-03-11T23:01:51.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6600 IOs, 2716497920 bytes 2024-03-11T23:01:51.511 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6700 IOs, 2762670080 bytes 2024-03-11T23:01:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6800 IOs, 2777647104 bytes 2024-03-11T23:01:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6900 IOs, 2816117760 bytes 2024-03-11T23:01:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7000 IOs, 2878220288 bytes 2024-03-11T23:01:52.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7100 IOs, 2916038656 bytes 2024-03-11T23:01:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7200 IOs, 3064535040 bytes 2024-03-11T23:01:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7300 IOs, 3108400128 bytes 2024-03-11T23:01:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7400 IOs, 3167698944 bytes 2024-03-11T23:01:53.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7500 IOs, 3183664128 bytes 2024-03-11T23:01:53.288 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7600 IOs, 3220332544 bytes 2024-03-11T23:01:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7700 IOs, 3234012160 bytes 2024-03-11T23:01:53.604 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7800 IOs, 3280422912 bytes 2024-03-11T23:01:53.830 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7900 IOs, 3353193472 bytes 2024-03-11T23:01:53.977 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:01:53.977 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:01:53.977 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:01:53.977 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:01:53.977 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:01:53.977 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_15 2024-03-11T23:01:53.977 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 snapshot 128M 2024-03-11T23:01:53.977 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:01:53.977 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:53.977 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:01:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_15 2024-03-11T23:01:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:01:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:01:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_15 128M 2024-03-11T23:01:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:01:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:01:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_15 2024-03-11T23:01:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:01:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:01:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:01:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:01:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_15 2024-03-11T23:01:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:01:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:01:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:01:54.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:01:54.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_15 snapshot 2024-03-11T23:01:54.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:01:54.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:01:54.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_15 2024-03-11T23:01:54.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:01:54.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_15 snapshot 2024-03-11T23:01:55.008 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:01:55.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_15 2024-03-11T23:01:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_15': 2024-03-11T23:01:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:01:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:01:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:01:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 12eac7f4a389 2024-03-11T23:01:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.12eac7f4a389 2024-03-11T23:01:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:01:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:01:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:01:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:01:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:01:54 2024 2024-03-11T23:01:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:01:54 2024 2024-03-11T23:01:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:01:54 2024 2024-03-11T23:01:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:01:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:01:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: bfe4c714-e6df-48ee-b9ba-69d0b7a5f35b 2024-03-11T23:01:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:01:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:01:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_15 2024-03-11T23:01:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:01:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:01:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_15 2024-03-11T23:01:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:01:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=8 2024-03-11T23:01:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:01:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:01:55.289 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 43878400 bytes 2024-03-11T23:01:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 86024192 bytes 2024-03-11T23:01:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 168243200 bytes 2024-03-11T23:01:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 185822208 bytes 2024-03-11T23:01:55.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 210607104 bytes 2024-03-11T23:01:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 271595520 bytes 2024-03-11T23:01:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 332033024 bytes 2024-03-11T23:01:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 357319680 bytes 2024-03-11T23:01:56.783 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 431259648 bytes 2024-03-11T23:01:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 511521792 bytes 2024-03-11T23:01:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 563961856 bytes 2024-03-11T23:01:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 565250048 bytes 2024-03-11T23:01:57.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 591195136 bytes 2024-03-11T23:01:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 659854336 bytes 2024-03-11T23:01:57.925 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 706072576 bytes 2024-03-11T23:01:58.181 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 783299584 bytes 2024-03-11T23:01:58.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 813079552 bytes 2024-03-11T23:01:58.531 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 834926592 bytes 2024-03-11T23:01:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 847144960 bytes 2024-03-11T23:01:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 882556928 bytes 2024-03-11T23:01:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 921223168 bytes 2024-03-11T23:01:59.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 1002567680 bytes 2024-03-11T23:01:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 1043978240 bytes 2024-03-11T23:01:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 1140235264 bytes 2024-03-11T23:02:00.092 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1283086336 bytes 2024-03-11T23:02:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1360848896 bytes 2024-03-11T23:02:00.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1440614400 bytes 2024-03-11T23:02:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1513741312 bytes 2024-03-11T23:02:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1547136000 bytes 2024-03-11T23:02:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1579350016 bytes 2024-03-11T23:02:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1642736640 bytes 2024-03-11T23:02:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1698397184 bytes 2024-03-11T23:02:01.791 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1754258432 bytes 2024-03-11T23:02:02.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1817772032 bytes 2024-03-11T23:02:02.121 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1845996544 bytes 2024-03-11T23:02:02.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1867942912 bytes 2024-03-11T23:02:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1900145664 bytes 2024-03-11T23:02:02.605 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1925486592 bytes 2024-03-11T23:02:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1948090368 bytes 2024-03-11T23:02:02.938 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 2002027520 bytes 2024-03-11T23:02:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:02:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:02:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:02:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:02:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:02:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_16 2024-03-11T23:02:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 snapshot 128M 2024-03-11T23:02:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:02:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_16 2024-03-11T23:02:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:02:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:02:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_16 128M 2024-03-11T23:02:03.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:02:03.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:03.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:03.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:03.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_16 2024-03-11T23:02:03.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:03.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:02:03.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:02:03.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:02:03.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:03.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_16 2024-03-11T23:02:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:02:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:02:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:02:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_16 snapshot 2024-03-11T23:02:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:02:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_16 2024-03-11T23:02:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:02:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_16 snapshot 2024-03-11T23:02:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:02:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_16 2024-03-11T23:02:04.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_16': 2024-03-11T23:02:04.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:02:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:02:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:02:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 12f326cfb0e1 2024-03-11T23:02:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.12f326cfb0e1 2024-03-11T23:02:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:02:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:02:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:02:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:02:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:02:03 2024 2024-03-11T23:02:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:02:03 2024 2024-03-11T23:02:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:02:03 2024 2024-03-11T23:02:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:02:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:02:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 8afe84ee-3bfc-43a7-b04d-7925c8799653 2024-03-11T23:02:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:02:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:02:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_16 2024-03-11T23:02:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:02:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_16 2024-03-11T23:02:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:02:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=6 2024-03-11T23:02:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:02:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:02:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 67636224 bytes 2024-03-11T23:02:04.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 77882368 bytes 2024-03-11T23:02:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 104794112 bytes 2024-03-11T23:02:04.845 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 164853760 bytes 2024-03-11T23:02:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 192214016 bytes 2024-03-11T23:02:05.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 233708544 bytes 2024-03-11T23:02:05.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 269288448 bytes 2024-03-11T23:02:05.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 304064512 bytes 2024-03-11T23:02:05.680 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 340004864 bytes 2024-03-11T23:02:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 446760960 bytes 2024-03-11T23:02:06.065 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 457913344 bytes 2024-03-11T23:02:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 508304384 bytes 2024-03-11T23:02:06.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 541751296 bytes 2024-03-11T23:02:06.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 572055552 bytes 2024-03-11T23:02:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 658656256 bytes 2024-03-11T23:02:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 701320192 bytes 2024-03-11T23:02:07.167 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 715173888 bytes 2024-03-11T23:02:07.265 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 736458752 bytes 2024-03-11T23:02:07.414 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 761918464 bytes 2024-03-11T23:02:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 776869888 bytes 2024-03-11T23:02:07.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 813956096 bytes 2024-03-11T23:02:07.867 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 846504960 bytes 2024-03-11T23:02:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 865534976 bytes 2024-03-11T23:02:08.260 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 950223872 bytes 2024-03-11T23:02:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1019912192 bytes 2024-03-11T23:02:08.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1089123328 bytes 2024-03-11T23:02:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1168474112 bytes 2024-03-11T23:02:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1262366720 bytes 2024-03-11T23:02:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1310701568 bytes 2024-03-11T23:02:09.512 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1316998144 bytes 2024-03-11T23:02:09.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1362030592 bytes 2024-03-11T23:02:09.913 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1422988288 bytes 2024-03-11T23:02:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1472593920 bytes 2024-03-11T23:02:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:02:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:02:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:02:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:02:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:02:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_17 2024-03-11T23:02:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 snapshot 128M 2024-03-11T23:02:10.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:02:10.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:10.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:10.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:10.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_17 2024-03-11T23:02:10.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:10.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:02:10.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:02:10.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:10.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_17 128M 2024-03-11T23:02:10.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:02:10.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:10.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:10.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_17 2024-03-11T23:02:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:02:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:02:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:02:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_17 2024-03-11T23:02:10.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:02:10.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:10.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:02:10.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:02:10.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_17 snapshot 2024-03-11T23:02:10.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:02:10.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:10.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_17 2024-03-11T23:02:10.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:02:10.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_17 snapshot 2024-03-11T23:02:11.095 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:02:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_17 2024-03-11T23:02:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_17': 2024-03-11T23:02:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:02:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:02:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:02:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 12fc48bc7729 2024-03-11T23:02:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.12fc48bc7729 2024-03-11T23:02:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:02:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:02:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:02:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:02:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:02:10 2024 2024-03-11T23:02:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:02:10 2024 2024-03-11T23:02:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:02:10 2024 2024-03-11T23:02:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:02:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:02:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 2a3f0f67-66fa-49bb-9d97-83ed1ad32bc2 2024-03-11T23:02:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:02:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:02:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_17 2024-03-11T23:02:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:02:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_17 2024-03-11T23:02:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:02:11.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=12 2024-03-11T23:02:11.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:02:11.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:02:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 85285888 bytes 2024-03-11T23:02:11.575 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 113140736 bytes 2024-03-11T23:02:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 176206848 bytes 2024-03-11T23:02:12.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 265482240 bytes 2024-03-11T23:02:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 312026112 bytes 2024-03-11T23:02:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 358492160 bytes 2024-03-11T23:02:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 359101440 bytes 2024-03-11T23:02:12.629 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 375808000 bytes 2024-03-11T23:02:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 430171136 bytes 2024-03-11T23:02:13.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 464961536 bytes 2024-03-11T23:02:13.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 466596864 bytes 2024-03-11T23:02:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 519906304 bytes 2024-03-11T23:02:13.468 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 542556160 bytes 2024-03-11T23:02:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 595113984 bytes 2024-03-11T23:02:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 675547136 bytes 2024-03-11T23:02:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 731164672 bytes 2024-03-11T23:02:14.429 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 854936576 bytes 2024-03-11T23:02:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 937845760 bytes 2024-03-11T23:02:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 999145472 bytes 2024-03-11T23:02:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 1105666048 bytes 2024-03-11T23:02:15.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 1128505344 bytes 2024-03-11T23:02:15.621 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 1213208576 bytes 2024-03-11T23:02:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 1229761536 bytes 2024-03-11T23:02:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 1350568960 bytes 2024-03-11T23:02:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1393010688 bytes 2024-03-11T23:02:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1457538048 bytes 2024-03-11T23:02:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1538122752 bytes 2024-03-11T23:02:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1560330240 bytes 2024-03-11T23:02:17.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1614326784 bytes 2024-03-11T23:02:17.272 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1663209472 bytes 2024-03-11T23:02:17.514 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1724561408 bytes 2024-03-11T23:02:17.682 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1757362176 bytes 2024-03-11T23:02:17.899 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1821168640 bytes 2024-03-11T23:02:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1892319232 bytes 2024-03-11T23:02:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1940571136 bytes 2024-03-11T23:02:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1965334528 bytes 2024-03-11T23:02:18.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 2005289984 bytes 2024-03-11T23:02:18.842 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 2055757824 bytes 2024-03-11T23:02:18.943 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 2061976576 bytes 2024-03-11T23:02:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 2080389120 bytes 2024-03-11T23:02:19.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 2085395456 bytes 2024-03-11T23:02:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 2162941952 bytes 2024-03-11T23:02:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 2217222144 bytes 2024-03-11T23:02:19.770 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 2243247104 bytes 2024-03-11T23:02:19.876 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 2243856384 bytes 2024-03-11T23:02:20.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 2277800960 bytes 2024-03-11T23:02:20.135 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 2278392832 bytes 2024-03-11T23:02:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 2317542400 bytes 2024-03-11T23:02:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 2337105920 bytes 2024-03-11T23:02:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 2391296000 bytes 2024-03-11T23:02:20.832 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5100 IOs, 2465451008 bytes 2024-03-11T23:02:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5200 IOs, 2502256640 bytes 2024-03-11T23:02:21.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5300 IOs, 2550739968 bytes 2024-03-11T23:02:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5400 IOs, 2604361728 bytes 2024-03-11T23:02:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5500 IOs, 2646781952 bytes 2024-03-11T23:02:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5600 IOs, 2651891712 bytes 2024-03-11T23:02:21.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5700 IOs, 2679541760 bytes 2024-03-11T23:02:22.014 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5800 IOs, 2722499584 bytes 2024-03-11T23:02:22.158 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5900 IOs, 2738272256 bytes 2024-03-11T23:02:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6000 IOs, 2783821824 bytes 2024-03-11T23:02:22.505 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6100 IOs, 2824216576 bytes 2024-03-11T23:02:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6200 IOs, 2856779776 bytes 2024-03-11T23:02:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6300 IOs, 2918702080 bytes 2024-03-11T23:02:23.036 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6400 IOs, 2946876416 bytes 2024-03-11T23:02:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:02:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:02:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:02:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:02:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:02:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_18 2024-03-11T23:02:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 snapshot 128M 2024-03-11T23:02:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:02:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_18 2024-03-11T23:02:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:02:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:02:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_18 128M 2024-03-11T23:02:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:02:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_18 2024-03-11T23:02:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:02:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:02:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:02:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_18 2024-03-11T23:02:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:02:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:02:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:02:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_18 snapshot 2024-03-11T23:02:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:02:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_18 2024-03-11T23:02:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:02:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_18 snapshot 2024-03-11T23:02:24.160 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:02:24.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_18 2024-03-11T23:02:24.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_18': 2024-03-11T23:02:24.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:02:24.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:02:24.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:02:24.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 130589f410a5 2024-03-11T23:02:24.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.130589f410a5 2024-03-11T23:02:24.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:02:24.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:02:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:02:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:02:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:02:23 2024 2024-03-11T23:02:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:02:23 2024 2024-03-11T23:02:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:02:23 2024 2024-03-11T23:02:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:02:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:02:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 387adbd8-be3f-401f-ba8e-7062c5a54dfd 2024-03-11T23:02:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:02:24.204 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:02:24.204 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_18 2024-03-11T23:02:24.204 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:02:24.204 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:24.204 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_18 2024-03-11T23:02:24.204 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:02:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=6 2024-03-11T23:02:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:02:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:02:24.431 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 42342400 bytes 2024-03-11T23:02:24.587 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 84049920 bytes 2024-03-11T23:02:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 132983808 bytes 2024-03-11T23:02:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 219389952 bytes 2024-03-11T23:02:25.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 255397888 bytes 2024-03-11T23:02:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 295085056 bytes 2024-03-11T23:02:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 309443584 bytes 2024-03-11T23:02:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 340875264 bytes 2024-03-11T23:02:25.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 394608640 bytes 2024-03-11T23:02:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 420923392 bytes 2024-03-11T23:02:26.111 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 446199808 bytes 2024-03-11T23:02:26.323 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 480415744 bytes 2024-03-11T23:02:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 526480384 bytes 2024-03-11T23:02:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 607262720 bytes 2024-03-11T23:02:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 663566336 bytes 2024-03-11T23:02:27.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 678164480 bytes 2024-03-11T23:02:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 714049536 bytes 2024-03-11T23:02:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 745534464 bytes 2024-03-11T23:02:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 792201216 bytes 2024-03-11T23:02:27.777 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 860662784 bytes 2024-03-11T23:02:28.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 948935680 bytes 2024-03-11T23:02:28.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 990434304 bytes 2024-03-11T23:02:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 1051943936 bytes 2024-03-11T23:02:28.570 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 1052554240 bytes 2024-03-11T23:02:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1091144704 bytes 2024-03-11T23:02:28.849 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1104670720 bytes 2024-03-11T23:02:28.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1126526976 bytes 2024-03-11T23:02:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1129830400 bytes 2024-03-11T23:02:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1158841344 bytes 2024-03-11T23:02:29.385 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1191999488 bytes 2024-03-11T23:02:29.662 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1285210112 bytes 2024-03-11T23:02:29.788 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1317748736 bytes 2024-03-11T23:02:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1393660928 bytes 2024-03-11T23:02:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:02:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:02:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:02:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:02:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:02:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_19 2024-03-11T23:02:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 snapshot 128M 2024-03-11T23:02:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:02:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_19 2024-03-11T23:02:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:02:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:02:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_19 128M 2024-03-11T23:02:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:02:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_19 2024-03-11T23:02:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:02:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:02:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:02:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_19 2024-03-11T23:02:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:02:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:02:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:02:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_19 snapshot 2024-03-11T23:02:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:02:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_19 2024-03-11T23:02:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:02:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_19 snapshot 2024-03-11T23:02:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:02:31.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_19 2024-03-11T23:02:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_19': 2024-03-11T23:02:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:02:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:02:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:02:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 130fc24052b7 2024-03-11T23:02:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.130fc24052b7 2024-03-11T23:02:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:02:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:02:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:02:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:02:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:02:30 2024 2024-03-11T23:02:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:02:30 2024 2024-03-11T23:02:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:02:30 2024 2024-03-11T23:02:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:02:31.269 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:02:31.269 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 06685589-ab88-4e65-be67-9272ee49920c 2024-03-11T23:02:31.269 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:02:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:02:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_19 2024-03-11T23:02:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:02:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_19 2024-03-11T23:02:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:02:31.273 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=5 2024-03-11T23:02:31.273 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:02:31.273 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:02:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 78138368 bytes 2024-03-11T23:02:31.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 164631552 bytes 2024-03-11T23:02:32.169 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 244722688 bytes 2024-03-11T23:02:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 268938240 bytes 2024-03-11T23:02:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 290051072 bytes 2024-03-11T23:02:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 359660544 bytes 2024-03-11T23:02:32.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 388222976 bytes 2024-03-11T23:02:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 434546688 bytes 2024-03-11T23:02:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 459807744 bytes 2024-03-11T23:02:33.383 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 501000192 bytes 2024-03-11T23:02:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 565407744 bytes 2024-03-11T23:02:33.749 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 620728320 bytes 2024-03-11T23:02:33.999 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 692550656 bytes 2024-03-11T23:02:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 744976384 bytes 2024-03-11T23:02:34.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 764569600 bytes 2024-03-11T23:02:34.591 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 843414528 bytes 2024-03-11T23:02:34.783 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 883265536 bytes 2024-03-11T23:02:34.942 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 908333056 bytes 2024-03-11T23:02:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 988342272 bytes 2024-03-11T23:02:35.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 1038637056 bytes 2024-03-11T23:02:35.663 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 1138471936 bytes 2024-03-11T23:02:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 1185486848 bytes 2024-03-11T23:02:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 1317706752 bytes 2024-03-11T23:02:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:02:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:02:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:02:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:02:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:02:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_20 2024-03-11T23:02:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 snapshot 128M 2024-03-11T23:02:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:02:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_20 2024-03-11T23:02:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:02:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:02:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_20 128M 2024-03-11T23:02:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:02:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_20 2024-03-11T23:02:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:02:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:02:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:02:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_20 2024-03-11T23:02:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:02:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:02:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:02:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_20 snapshot 2024-03-11T23:02:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:02:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_20 2024-03-11T23:02:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:02:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_20 snapshot 2024-03-11T23:02:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:02:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_20 2024-03-11T23:02:37.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_20': 2024-03-11T23:02:37.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:02:37.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:02:37.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:02:37.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 1317264a0338 2024-03-11T23:02:37.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.1317264a0338 2024-03-11T23:02:37.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:02:37.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:02:37.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:02:37.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:02:37.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:02:36 2024 2024-03-11T23:02:37.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:02:36 2024 2024-03-11T23:02:37.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:02:36 2024 2024-03-11T23:02:37.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:02:37.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:02:37.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 7b95035b-6c99-40a9-947e-f623904f126b 2024-03-11T23:02:37.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:02:37.317 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:02:37.317 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_20 2024-03-11T23:02:37.317 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:02:37.317 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:37.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_20 2024-03-11T23:02:37.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:02:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=5 2024-03-11T23:02:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:02:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:02:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 120373248 bytes 2024-03-11T23:02:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 153380864 bytes 2024-03-11T23:02:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 157888512 bytes 2024-03-11T23:02:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 225007616 bytes 2024-03-11T23:02:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 292141056 bytes 2024-03-11T23:02:38.523 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 316645376 bytes 2024-03-11T23:02:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 341756928 bytes 2024-03-11T23:02:38.784 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 358646784 bytes 2024-03-11T23:02:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 408578048 bytes 2024-03-11T23:02:39.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 479132672 bytes 2024-03-11T23:02:39.419 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 520346624 bytes 2024-03-11T23:02:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 533003264 bytes 2024-03-11T23:02:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 581334016 bytes 2024-03-11T23:02:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 589372416 bytes 2024-03-11T23:02:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 634117120 bytes 2024-03-11T23:02:40.309 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 719503360 bytes 2024-03-11T23:02:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 791160832 bytes 2024-03-11T23:02:40.672 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 808384512 bytes 2024-03-11T23:02:40.900 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 877068288 bytes 2024-03-11T23:02:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 924812288 bytes 2024-03-11T23:02:41.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 932842496 bytes 2024-03-11T23:02:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 978202624 bytes 2024-03-11T23:02:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 1014036480 bytes 2024-03-11T23:02:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 1076140032 bytes 2024-03-11T23:02:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1125328896 bytes 2024-03-11T23:02:42.122 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1168104448 bytes 2024-03-11T23:02:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1199240192 bytes 2024-03-11T23:02:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:02:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:02:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:02:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:02:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:02:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_21 2024-03-11T23:02:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 snapshot 128M 2024-03-11T23:02:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:02:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:42.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:42.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:42.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_21 2024-03-11T23:02:42.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:42.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:02:42.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:02:42.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:42.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_21 128M 2024-03-11T23:02:42.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:02:42.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:42.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:42.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:42.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_21 2024-03-11T23:02:42.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:42.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:02:42.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:02:42.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:02:42.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:42.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_21 2024-03-11T23:02:42.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:02:42.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:42.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:02:42.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:02:42.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_21 snapshot 2024-03-11T23:02:42.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:02:42.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:42.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_21 2024-03-11T23:02:42.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:02:42.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_21 snapshot 2024-03-11T23:02:43.302 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:02:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_21 2024-03-11T23:02:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_21': 2024-03-11T23:02:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:02:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:02:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:02:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 131f3115326c 2024-03-11T23:02:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.131f3115326c 2024-03-11T23:02:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:02:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:02:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:02:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:02:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:02:42 2024 2024-03-11T23:02:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:02:42 2024 2024-03-11T23:02:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:02:42 2024 2024-03-11T23:02:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:02:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:02:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 32652090-5ba2-4262-a3f3-521e042b1019 2024-03-11T23:02:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:02:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:02:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_21 2024-03-11T23:02:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:02:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_21 2024-03-11T23:02:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:02:43.352 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=8 2024-03-11T23:02:43.352 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:02:43.352 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:02:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 1050624 bytes 2024-03-11T23:02:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 47206400 bytes 2024-03-11T23:02:43.787 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4679572514137258) > weight (0.25) 2024-03-11T23:02:43.788 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7682743104909978) > weight (0.25) 2024-03-11T23:02:43.788 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-03-11T23:02:43.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-03-11T23:02:43.788 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-03-11T23:02:43.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi132.stderr:2024-03-11T23:02:43.786+0000 7fc1ba818640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 40107) UID: 1000 2024-03-11T23:02:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 101994496 bytes 2024-03-11T23:02:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 147716096 bytes 2024-03-11T23:02:44.167 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 153323520 bytes 2024-03-11T23:02:44.316 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 179006464 bytes 2024-03-11T23:02:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 245771264 bytes 2024-03-11T23:02:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 289280000 bytes 2024-03-11T23:02:44.789 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-03-11T23:02:44.789 INFO:teuthology.orchestra.run:waiting for 600 2024-03-11T23:02:44.789 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-03-11T23:02:44.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-03-11T23:02:44.789 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-03-11T23:02:44.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-03-11T23:02:44.792 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2024-03-11T23:02:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 296202240 bytes 2024-03-11T23:02:44.996 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 333521920 bytes 2024-03-11T23:02:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 360850432 bytes 2024-03-11T23:02:45.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 412063744 bytes 2024-03-11T23:02:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 527682560 bytes 2024-03-11T23:02:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 556471296 bytes 2024-03-11T23:02:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 643380224 bytes 2024-03-11T23:02:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 690951168 bytes 2024-03-11T23:02:46.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 699106304 bytes 2024-03-11T23:02:46.571 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 720271360 bytes 2024-03-11T23:02:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 720893952 bytes 2024-03-11T23:02:46.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 761101312 bytes 2024-03-11T23:02:47.040 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 823979008 bytes 2024-03-11T23:02:47.377 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 951470080 bytes 2024-03-11T23:02:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 952077312 bytes 2024-03-11T23:02:47.650 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 972600320 bytes 2024-03-11T23:02:47.780 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 995608576 bytes 2024-03-11T23:02:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1048609792 bytes 2024-03-11T23:02:48.080 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1057766400 bytes 2024-03-11T23:02:48.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1084295168 bytes 2024-03-11T23:02:48.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1112521728 bytes 2024-03-11T23:02:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1156033536 bytes 2024-03-11T23:02:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1243925504 bytes 2024-03-11T23:02:49.058 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1296108544 bytes 2024-03-11T23:02:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1380349952 bytes 2024-03-11T23:02:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1452646400 bytes 2024-03-11T23:02:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1466061824 bytes 2024-03-11T23:02:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1527442432 bytes 2024-03-11T23:02:50.222 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1546768384 bytes 2024-03-11T23:02:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1568092160 bytes 2024-03-11T23:02:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1639837696 bytes 2024-03-11T23:02:50.860 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1733149696 bytes 2024-03-11T23:02:51.021 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1758651392 bytes 2024-03-11T23:02:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1783479296 bytes 2024-03-11T23:02:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1786859520 bytes 2024-03-11T23:02:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:02:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:02:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:02:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:02:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:02:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_22 2024-03-11T23:02:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 snapshot 128M 2024-03-11T23:02:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:02:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_22 2024-03-11T23:02:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:02:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:02:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_22 128M 2024-03-11T23:02:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:02:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_22 2024-03-11T23:02:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:02:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:02:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:02:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_22 2024-03-11T23:02:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:02:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:02:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:02:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_22 snapshot 2024-03-11T23:02:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:02:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_22 2024-03-11T23:02:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:02:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_22 snapshot 2024-03-11T23:02:51.843 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:02:51.849 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_22 2024-03-11T23:02:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_22': 2024-03-11T23:02:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:02:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:02:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:02:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 1332c35ddb4f 2024-03-11T23:02:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.1332c35ddb4f 2024-03-11T23:02:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:02:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:02:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:02:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:02:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:02:51 2024 2024-03-11T23:02:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:02:51 2024 2024-03-11T23:02:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:02:51 2024 2024-03-11T23:02:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:02:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:02:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 6b2da8c4-e5c3-4ec9-acc6-083f710996ff 2024-03-11T23:02:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:02:51.888 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:02:51.888 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_22 2024-03-11T23:02:51.888 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:02:51.888 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:51.888 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_22 2024-03-11T23:02:51.889 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:02:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=7 2024-03-11T23:02:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:02:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:02:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 15436800 bytes 2024-03-11T23:02:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 37921792 bytes 2024-03-11T23:02:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 73499648 bytes 2024-03-11T23:02:52.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 75438080 bytes 2024-03-11T23:02:52.794 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 132166656 bytes 2024-03-11T23:02:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 184828928 bytes 2024-03-11T23:02:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 268595200 bytes 2024-03-11T23:02:53.464 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 336563200 bytes 2024-03-11T23:02:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 399990784 bytes 2024-03-11T23:02:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 491275264 bytes 2024-03-11T23:02:54.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 516653056 bytes 2024-03-11T23:02:54.310 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 557988864 bytes 2024-03-11T23:02:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 642889728 bytes 2024-03-11T23:02:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 706396160 bytes 2024-03-11T23:02:54.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 757685248 bytes 2024-03-11T23:02:55.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 791677952 bytes 2024-03-11T23:02:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 814783488 bytes 2024-03-11T23:02:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 858589184 bytes 2024-03-11T23:02:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 908508160 bytes 2024-03-11T23:02:56.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 968096768 bytes 2024-03-11T23:02:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 1042216960 bytes 2024-03-11T23:02:56.511 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 1086523392 bytes 2024-03-11T23:02:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 1154782208 bytes 2024-03-11T23:02:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 1203519488 bytes 2024-03-11T23:02:57.113 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1244208128 bytes 2024-03-11T23:02:57.307 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1287968768 bytes 2024-03-11T23:02:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1293858816 bytes 2024-03-11T23:02:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1347516416 bytes 2024-03-11T23:02:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1382709248 bytes 2024-03-11T23:02:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1446685696 bytes 2024-03-11T23:02:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1474461696 bytes 2024-03-11T23:02:58.251 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1494443008 bytes 2024-03-11T23:02:58.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1550403584 bytes 2024-03-11T23:02:58.563 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1551021056 bytes 2024-03-11T23:02:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1615432704 bytes 2024-03-11T23:02:58.896 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:02:58.896 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:02:58.896 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:02:58.896 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:02:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:02:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_23 2024-03-11T23:02:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 snapshot 128M 2024-03-11T23:02:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:02:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_23 2024-03-11T23:02:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:02:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:02:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_23 128M 2024-03-11T23:02:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:02:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_23 2024-03-11T23:02:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:02:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:02:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:02:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:02:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_23 2024-03-11T23:02:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:02:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:02:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:02:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_23 snapshot 2024-03-11T23:02:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:02:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_23 2024-03-11T23:02:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:02:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_23 snapshot 2024-03-11T23:02:59.900 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:02:59.906 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_23 2024-03-11T23:02:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_23': 2024-03-11T23:02:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:02:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:02:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:02:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 133a32a5ff37 2024-03-11T23:02:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.133a32a5ff37 2024-03-11T23:02:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:02:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:02:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:02:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:02:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:02:58 2024 2024-03-11T23:02:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:02:58 2024 2024-03-11T23:02:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:02:58 2024 2024-03-11T23:02:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:02:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:02:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 6f278f19-20af-4e77-81d1-1dfe17bc389b 2024-03-11T23:02:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:02:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:02:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_23 2024-03-11T23:02:59.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:02:59.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:02:59.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_23 2024-03-11T23:02:59.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:02:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=10 2024-03-11T23:02:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:02:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:03:00.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 30118912 bytes 2024-03-11T23:03:00.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 96574464 bytes 2024-03-11T23:03:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 149449728 bytes 2024-03-11T23:03:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 189189120 bytes 2024-03-11T23:03:00.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 242611200 bytes 2024-03-11T23:03:01.160 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 338248704 bytes 2024-03-11T23:03:01.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 424033280 bytes 2024-03-11T23:03:01.634 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 448147456 bytes 2024-03-11T23:03:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 454319104 bytes 2024-03-11T23:03:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 496199680 bytes 2024-03-11T23:03:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 574844928 bytes 2024-03-11T23:03:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 622748672 bytes 2024-03-11T23:03:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 658463744 bytes 2024-03-11T23:03:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 675125248 bytes 2024-03-11T23:03:02.799 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 717390848 bytes 2024-03-11T23:03:03.031 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 789181440 bytes 2024-03-11T23:03:03.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 870017024 bytes 2024-03-11T23:03:03.473 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 915503104 bytes 2024-03-11T23:03:03.656 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 961973248 bytes 2024-03-11T23:03:03.821 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 1014566912 bytes 2024-03-11T23:03:04.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 1113576448 bytes 2024-03-11T23:03:04.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 1178209280 bytes 2024-03-11T23:03:04.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 1246696448 bytes 2024-03-11T23:03:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 1248785408 bytes 2024-03-11T23:03:04.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1319995392 bytes 2024-03-11T23:03:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1359105024 bytes 2024-03-11T23:03:05.254 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1389564928 bytes 2024-03-11T23:03:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1450544128 bytes 2024-03-11T23:03:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1477804032 bytes 2024-03-11T23:03:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1480382464 bytes 2024-03-11T23:03:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1506515968 bytes 2024-03-11T23:03:06.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1552656384 bytes 2024-03-11T23:03:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1586886656 bytes 2024-03-11T23:03:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1640889344 bytes 2024-03-11T23:03:06.586 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1661917184 bytes 2024-03-11T23:03:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1671998464 bytes 2024-03-11T23:03:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1737273344 bytes 2024-03-11T23:03:07.080 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1772849152 bytes 2024-03-11T23:03:07.224 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1796461568 bytes 2024-03-11T23:03:07.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1797063680 bytes 2024-03-11T23:03:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1852334080 bytes 2024-03-11T23:03:07.690 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1891061760 bytes 2024-03-11T23:03:07.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1974804480 bytes 2024-03-11T23:03:08.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 2014665728 bytes 2024-03-11T23:03:08.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 2062584832 bytes 2024-03-11T23:03:08.696 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 2200829952 bytes 2024-03-11T23:03:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 2233601024 bytes 2024-03-11T23:03:09.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 2259946496 bytes 2024-03-11T23:03:09.219 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 2314398720 bytes 2024-03-11T23:03:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 2342063104 bytes 2024-03-11T23:03:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5100 IOs, 2395456512 bytes 2024-03-11T23:03:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5200 IOs, 2443808768 bytes 2024-03-11T23:03:09.957 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:03:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:03:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:03:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:03:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:03:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_24 2024-03-11T23:03:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 snapshot 128M 2024-03-11T23:03:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:03:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:03:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_24 2024-03-11T23:03:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:03:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:03:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_24 128M 2024-03-11T23:03:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:03:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:03:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_24 2024-03-11T23:03:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:03:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:03:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:03:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_24 2024-03-11T23:03:10.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:03:10.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:03:10.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:03:10.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:03:10.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_24 snapshot 2024-03-11T23:03:10.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:03:10.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:03:10.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_24 2024-03-11T23:03:10.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:03:10.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_24 snapshot 2024-03-11T23:03:11.008 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:03:11.014 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_24 2024-03-11T23:03:11.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_24': 2024-03-11T23:03:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:03:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:03:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:03:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 1342c6224d7 2024-03-11T23:03:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.1342c6224d7 2024-03-11T23:03:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:03:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:03:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:03:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:03:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:03:09 2024 2024-03-11T23:03:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:03:09 2024 2024-03-11T23:03:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:03:09 2024 2024-03-11T23:03:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:03:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:03:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: edbba68c-29ef-4285-9e1b-327ac0002f9c 2024-03-11T23:03:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:03:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:03:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_24 2024-03-11T23:03:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:03:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:03:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_24 2024-03-11T23:03:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:03:11.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=8 2024-03-11T23:03:11.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:03:11.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:03:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 22480896 bytes 2024-03-11T23:03:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 29934592 bytes 2024-03-11T23:03:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 62216192 bytes 2024-03-11T23:03:11.618 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 76825600 bytes 2024-03-11T23:03:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 171161600 bytes 2024-03-11T23:03:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 245746688 bytes 2024-03-11T23:03:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 319379456 bytes 2024-03-11T23:03:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 327683072 bytes 2024-03-11T23:03:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 375408640 bytes 2024-03-11T23:03:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 386711552 bytes 2024-03-11T23:03:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 387296256 bytes 2024-03-11T23:03:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 387941376 bytes 2024-03-11T23:03:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 400922624 bytes 2024-03-11T23:03:13.222 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 413197312 bytes 2024-03-11T23:03:13.467 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 486129664 bytes 2024-03-11T23:03:13.609 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 510402560 bytes 2024-03-11T23:03:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 563147776 bytes 2024-03-11T23:03:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 582136832 bytes 2024-03-11T23:03:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 620256256 bytes 2024-03-11T23:03:14.288 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 656201728 bytes 2024-03-11T23:03:14.467 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 700603392 bytes 2024-03-11T23:03:14.566 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 701280256 bytes 2024-03-11T23:03:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 757267456 bytes 2024-03-11T23:03:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 836194304 bytes 2024-03-11T23:03:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 902080512 bytes 2024-03-11T23:03:15.432 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 947959808 bytes 2024-03-11T23:03:15.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1026490368 bytes 2024-03-11T23:03:15.841 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1061675008 bytes 2024-03-11T23:03:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1108687872 bytes 2024-03-11T23:03:16.253 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1171398656 bytes 2024-03-11T23:03:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1201585152 bytes 2024-03-11T23:03:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1217452032 bytes 2024-03-11T23:03:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1257866240 bytes 2024-03-11T23:03:16.868 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1277986816 bytes 2024-03-11T23:03:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1318024192 bytes 2024-03-11T23:03:17.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1345361920 bytes 2024-03-11T23:03:17.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1396825088 bytes 2024-03-11T23:03:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1433498624 bytes 2024-03-11T23:03:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1434083328 bytes 2024-03-11T23:03:17.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1471630336 bytes 2024-03-11T23:03:17.981 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1501084672 bytes 2024-03-11T23:03:18.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1607665664 bytes 2024-03-11T23:03:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1706381312 bytes 2024-03-11T23:03:18.760 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 1735881728 bytes 2024-03-11T23:03:18.926 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 1777686528 bytes 2024-03-11T23:03:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 1786147840 bytes 2024-03-11T23:03:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:03:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:03:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:03:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:03:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:03:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_25 2024-03-11T23:03:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 snapshot 128M 2024-03-11T23:03:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:03:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:03:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_25 2024-03-11T23:03:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:03:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:03:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_25 128M 2024-03-11T23:03:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:03:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:03:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_25 2024-03-11T23:03:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:03:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:03:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:03:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_25 2024-03-11T23:03:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:03:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:03:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:03:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:03:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_25 snapshot 2024-03-11T23:03:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:03:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:03:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_25 2024-03-11T23:03:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:03:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_25 snapshot 2024-03-11T23:03:20.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:03:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_25 2024-03-11T23:03:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_25': 2024-03-11T23:03:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:03:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:03:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:03:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 134bd6b6b6d4 2024-03-11T23:03:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.134bd6b6b6d4 2024-03-11T23:03:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:03:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:03:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:03:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:03:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:03:19 2024 2024-03-11T23:03:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:03:19 2024 2024-03-11T23:03:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:03:19 2024 2024-03-11T23:03:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:03:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:03:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: d58404ce-fb8c-4239-ba38-049eab7f80aa 2024-03-11T23:03:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:03:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:03:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_25 2024-03-11T23:03:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:03:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:03:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_25 2024-03-11T23:03:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:03:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=11 2024-03-11T23:03:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:03:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:03:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 54870016 bytes 2024-03-11T23:03:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 68453376 bytes 2024-03-11T23:03:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 103733248 bytes 2024-03-11T23:03:20.737 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 125936640 bytes 2024-03-11T23:03:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 158795776 bytes 2024-03-11T23:03:21.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 189423616 bytes 2024-03-11T23:03:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 230123520 bytes 2024-03-11T23:03:21.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 299764736 bytes 2024-03-11T23:03:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 384808960 bytes 2024-03-11T23:03:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 441272320 bytes 2024-03-11T23:03:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 516390912 bytes 2024-03-11T23:03:22.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 543372288 bytes 2024-03-11T23:03:22.488 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 579240960 bytes 2024-03-11T23:03:22.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 614121472 bytes 2024-03-11T23:03:22.849 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 666102784 bytes 2024-03-11T23:03:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 709435392 bytes 2024-03-11T23:03:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 777095168 bytes 2024-03-11T23:03:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 791088128 bytes 2024-03-11T23:03:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 815542272 bytes 2024-03-11T23:03:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 851234816 bytes 2024-03-11T23:03:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 915892224 bytes 2024-03-11T23:03:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 1015230464 bytes 2024-03-11T23:03:24.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 1047201792 bytes 2024-03-11T23:03:24.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 1112427520 bytes 2024-03-11T23:03:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1155428352 bytes 2024-03-11T23:03:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1156066304 bytes 2024-03-11T23:03:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1203031040 bytes 2024-03-11T23:03:25.136 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr:2024-03-11T23:03:25.132+0000 7fc25fa7a640 -1 reset not still connected to 0x55a87ea9dd40 2024-03-11T23:03:25.169 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1213512704 bytes 2024-03-11T23:03:25.375 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1266285568 bytes 2024-03-11T23:03:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1294793728 bytes 2024-03-11T23:03:25.692 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-03-11T23:03:25.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-03-11T23:03:25.692 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-03-11T23:03:25.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi132.stderr:2024-03-11T23:03:25.692+0000 7f8a88fc4640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 49811) UID: 1000 2024-03-11T23:03:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1348715520 bytes 2024-03-11T23:03:25.871 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1374943232 bytes 2024-03-11T23:03:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1420824576 bytes 2024-03-11T23:03:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1480671232 bytes 2024-03-11T23:03:26.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1514062848 bytes 2024-03-11T23:03:26.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1540335616 bytes 2024-03-11T23:03:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1542297600 bytes 2024-03-11T23:03:26.693 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-03-11T23:03:26.693 INFO:teuthology.orchestra.run:waiting for 600 2024-03-11T23:03:26.693 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-03-11T23:03:26.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-03-11T23:03:26.694 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-03-11T23:03:26.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-03-11T23:03:26.696 INFO:tasks.rbd_mirror_thrash:waiting for 99 secs before thrashing 2024-03-11T23:03:26.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1593136128 bytes 2024-03-11T23:03:27.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1626264576 bytes 2024-03-11T23:03:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1680406528 bytes 2024-03-11T23:03:27.404 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1717989376 bytes 2024-03-11T23:03:27.624 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1774586880 bytes 2024-03-11T23:03:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1795487744 bytes 2024-03-11T23:03:27.934 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 1829345280 bytes 2024-03-11T23:03:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 1887306752 bytes 2024-03-11T23:03:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 1919503360 bytes 2024-03-11T23:03:28.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 1940718592 bytes 2024-03-11T23:03:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 1991920640 bytes 2024-03-11T23:03:28.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 2096224256 bytes 2024-03-11T23:03:29.169 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 2170927104 bytes 2024-03-11T23:03:29.410 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5100 IOs, 2235518976 bytes 2024-03-11T23:03:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5200 IOs, 2347956224 bytes 2024-03-11T23:03:29.838 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5300 IOs, 2348548096 bytes 2024-03-11T23:03:30.114 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5400 IOs, 2439397376 bytes 2024-03-11T23:03:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5500 IOs, 2549134336 bytes 2024-03-11T23:03:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5600 IOs, 2620617728 bytes 2024-03-11T23:03:30.763 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5700 IOs, 2649295872 bytes 2024-03-11T23:03:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5800 IOs, 2706409472 bytes 2024-03-11T23:03:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:03:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:03:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:03:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:03:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:03:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_26 2024-03-11T23:03:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 snapshot 128M 2024-03-11T23:03:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:03:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:03:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_26 2024-03-11T23:03:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:03:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:03:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_26 128M 2024-03-11T23:03:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:03:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:03:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_26 2024-03-11T23:03:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:03:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:03:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:03:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_26 2024-03-11T23:03:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:03:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:03:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:03:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:03:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_26 snapshot 2024-03-11T23:03:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:03:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:03:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_26 2024-03-11T23:03:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:03:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_26 snapshot 2024-03-11T23:03:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:03:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_26 2024-03-11T23:03:32.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_26': 2024-03-11T23:03:32.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:03:32.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:03:32.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:03:32.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 135aec7a5d87 2024-03-11T23:03:32.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.135aec7a5d87 2024-03-11T23:03:32.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:03:32.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:03:32.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:03:32.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:03:32.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:03:31 2024 2024-03-11T23:03:32.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:03:31 2024 2024-03-11T23:03:32.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:03:31 2024 2024-03-11T23:03:32.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:03:32.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:03:32.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: d759a618-182e-4efe-82fa-def08c221d57 2024-03-11T23:03:32.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:03:32.159 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:03:32.159 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_26 2024-03-11T23:03:32.159 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:03:32.159 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:03:32.159 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_26 2024-03-11T23:03:32.159 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:03:32.160 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=9 2024-03-11T23:03:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:03:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:03:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 33465344 bytes 2024-03-11T23:03:32.586 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 99818496 bytes 2024-03-11T23:03:32.766 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 141541376 bytes 2024-03-11T23:03:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 193132544 bytes 2024-03-11T23:03:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 235821056 bytes 2024-03-11T23:03:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 248855552 bytes 2024-03-11T23:03:33.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 263468032 bytes 2024-03-11T23:03:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 288439296 bytes 2024-03-11T23:03:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 304876544 bytes 2024-03-11T23:03:33.818 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 345681920 bytes 2024-03-11T23:03:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 353526784 bytes 2024-03-11T23:03:34.074 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 386454528 bytes 2024-03-11T23:03:34.298 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 446788608 bytes 2024-03-11T23:03:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 557325312 bytes 2024-03-11T23:03:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 588853248 bytes 2024-03-11T23:03:34.907 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 599201792 bytes 2024-03-11T23:03:35.026 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 625629184 bytes 2024-03-11T23:03:35.157 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 631841792 bytes 2024-03-11T23:03:35.393 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 694876160 bytes 2024-03-11T23:03:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 761311232 bytes 2024-03-11T23:03:35.724 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 761932800 bytes 2024-03-11T23:03:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 816705536 bytes 2024-03-11T23:03:36.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 825762816 bytes 2024-03-11T23:03:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 869738496 bytes 2024-03-11T23:03:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 897320960 bytes 2024-03-11T23:03:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 925475840 bytes 2024-03-11T23:03:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 949360640 bytes 2024-03-11T23:03:36.901 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1017025536 bytes 2024-03-11T23:03:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1061025792 bytes 2024-03-11T23:03:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1070867456 bytes 2024-03-11T23:03:37.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1163496448 bytes 2024-03-11T23:03:37.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1252325376 bytes 2024-03-11T23:03:38.158 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1294774272 bytes 2024-03-11T23:03:38.258 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1295389696 bytes 2024-03-11T23:03:38.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1329739776 bytes 2024-03-11T23:03:38.679 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1406993408 bytes 2024-03-11T23:03:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1473209344 bytes 2024-03-11T23:03:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1514077184 bytes 2024-03-11T23:03:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1560005632 bytes 2024-03-11T23:03:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1571818496 bytes 2024-03-11T23:03:39.569 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1591554048 bytes 2024-03-11T23:03:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1608466432 bytes 2024-03-11T23:03:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1622383616 bytes 2024-03-11T23:03:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 1629806592 bytes 2024-03-11T23:03:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 1651999744 bytes 2024-03-11T23:03:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 1729137664 bytes 2024-03-11T23:03:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 1802632192 bytes 2024-03-11T23:03:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 1879529472 bytes 2024-03-11T23:03:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 1887526912 bytes 2024-03-11T23:03:41.167 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:03:41.167 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:03:41.167 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:03:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:03:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:03:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_27 2024-03-11T23:03:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 snapshot 128M 2024-03-11T23:03:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:03:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:03:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_27 2024-03-11T23:03:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:03:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:03:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_27 128M 2024-03-11T23:03:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:03:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:03:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_27 2024-03-11T23:03:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:03:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:03:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:03:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_27 2024-03-11T23:03:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:03:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:03:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:03:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:03:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_27 snapshot 2024-03-11T23:03:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:03:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:03:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_27 2024-03-11T23:03:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:03:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_27 snapshot 2024-03-11T23:03:42.131 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:03:42.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_27 2024-03-11T23:03:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_27': 2024-03-11T23:03:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:03:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:03:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:03:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 136330ab0c15 2024-03-11T23:03:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.136330ab0c15 2024-03-11T23:03:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:03:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:03:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:03:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:03:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:03:41 2024 2024-03-11T23:03:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:03:41 2024 2024-03-11T23:03:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:03:41 2024 2024-03-11T23:03:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:03:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:03:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: f55425aa-fe4e-4d48-89cf-caafa49c5423 2024-03-11T23:03:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:03:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:03:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_27 2024-03-11T23:03:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:03:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:03:42.178 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_27 2024-03-11T23:03:42.178 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:03:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=13 2024-03-11T23:03:42.180 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:03:42.180 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:03:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 47102976 bytes 2024-03-11T23:03:42.602 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 90084352 bytes 2024-03-11T23:03:42.790 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 133968896 bytes 2024-03-11T23:03:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 256688128 bytes 2024-03-11T23:03:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 307307520 bytes 2024-03-11T23:03:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 397740032 bytes 2024-03-11T23:03:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 404830208 bytes 2024-03-11T23:03:43.967 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 467504128 bytes 2024-03-11T23:03:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 514293760 bytes 2024-03-11T23:03:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 554738688 bytes 2024-03-11T23:03:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 611187712 bytes 2024-03-11T23:03:44.691 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 629252096 bytes 2024-03-11T23:03:44.899 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 696061952 bytes 2024-03-11T23:03:45.081 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 731469824 bytes 2024-03-11T23:03:45.227 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 753649664 bytes 2024-03-11T23:03:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 769180672 bytes 2024-03-11T23:03:45.615 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 864068608 bytes 2024-03-11T23:03:45.837 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 927316992 bytes 2024-03-11T23:03:45.999 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 959694848 bytes 2024-03-11T23:03:46.260 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 1044903936 bytes 2024-03-11T23:03:46.491 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 1110506496 bytes 2024-03-11T23:03:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 1147340800 bytes 2024-03-11T23:03:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 1155819520 bytes 2024-03-11T23:03:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 1196308480 bytes 2024-03-11T23:03:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1204345856 bytes 2024-03-11T23:03:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1268884480 bytes 2024-03-11T23:03:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1316328448 bytes 2024-03-11T23:03:47.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1374715904 bytes 2024-03-11T23:03:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1404751872 bytes 2024-03-11T23:03:47.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1425679360 bytes 2024-03-11T23:03:48.181 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1480222720 bytes 2024-03-11T23:03:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1523850240 bytes 2024-03-11T23:03:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1608251392 bytes 2024-03-11T23:03:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1608858624 bytes 2024-03-11T23:03:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1618193408 bytes 2024-03-11T23:03:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1638771712 bytes 2024-03-11T23:03:49.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1675207680 bytes 2024-03-11T23:03:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1718072320 bytes 2024-03-11T23:03:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1744739328 bytes 2024-03-11T23:03:49.634 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1787419648 bytes 2024-03-11T23:03:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1829048320 bytes 2024-03-11T23:03:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1856715776 bytes 2024-03-11T23:03:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1878631424 bytes 2024-03-11T23:03:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 1938242560 bytes 2024-03-11T23:03:50.507 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 1968065536 bytes 2024-03-11T23:03:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 1997080576 bytes 2024-03-11T23:03:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 2033161216 bytes 2024-03-11T23:03:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 2036575232 bytes 2024-03-11T23:03:51.068 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 2057674752 bytes 2024-03-11T23:03:51.302 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 2131278848 bytes 2024-03-11T23:03:51.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5100 IOs, 2156369920 bytes 2024-03-11T23:03:51.672 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5200 IOs, 2209897472 bytes 2024-03-11T23:03:51.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5300 IOs, 2296723456 bytes 2024-03-11T23:03:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5400 IOs, 2321819648 bytes 2024-03-11T23:03:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5500 IOs, 2378606592 bytes 2024-03-11T23:03:52.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5600 IOs, 2450059264 bytes 2024-03-11T23:03:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5700 IOs, 2502480896 bytes 2024-03-11T23:03:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5800 IOs, 2522277888 bytes 2024-03-11T23:03:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5900 IOs, 2539505664 bytes 2024-03-11T23:03:53.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6000 IOs, 2540123136 bytes 2024-03-11T23:03:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6100 IOs, 2619011072 bytes 2024-03-11T23:03:53.670 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6200 IOs, 2667761664 bytes 2024-03-11T23:03:54.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6300 IOs, 2773679104 bytes 2024-03-11T23:03:54.160 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6400 IOs, 2788809728 bytes 2024-03-11T23:03:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6500 IOs, 2884164608 bytes 2024-03-11T23:03:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6600 IOs, 2942527488 bytes 2024-03-11T23:03:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6700 IOs, 2969143296 bytes 2024-03-11T23:03:55.066 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6800 IOs, 3023095808 bytes 2024-03-11T23:03:55.184 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:03:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:03:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:03:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:03:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:03:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_28 2024-03-11T23:03:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 snapshot 128M 2024-03-11T23:03:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:03:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:03:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_28 2024-03-11T23:03:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:03:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:03:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_28 128M 2024-03-11T23:03:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:03:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:03:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_28 2024-03-11T23:03:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:03:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:03:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:03:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:03:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_28 2024-03-11T23:03:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:03:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:03:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:03:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:03:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_28 snapshot 2024-03-11T23:03:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:03:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:03:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_28 2024-03-11T23:03:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:03:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_28 snapshot 2024-03-11T23:03:55.822 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:03:55.827 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_28 2024-03-11T23:03:55.862 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_28': 2024-03-11T23:03:55.862 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:03:55.862 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:03:55.862 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:03:55.862 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 136c241b5810 2024-03-11T23:03:55.862 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.136c241b5810 2024-03-11T23:03:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:03:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:03:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:03:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:03:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:03:55 2024 2024-03-11T23:03:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:03:55 2024 2024-03-11T23:03:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:03:55 2024 2024-03-11T23:03:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:03:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:03:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 93b4a8e6-8dee-44ce-b32e-2d1497b47e9c 2024-03-11T23:03:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:03:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:03:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_28 2024-03-11T23:03:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:03:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:03:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_28 2024-03-11T23:03:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:03:55.867 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=11 2024-03-11T23:03:55.867 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:03:55.867 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:03:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 27801600 bytes 2024-03-11T23:03:56.383 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 123589632 bytes 2024-03-11T23:03:56.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 194929664 bytes 2024-03-11T23:03:56.796 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 228098048 bytes 2024-03-11T23:03:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 257204224 bytes 2024-03-11T23:03:57.134 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 285893632 bytes 2024-03-11T23:03:57.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 338249728 bytes 2024-03-11T23:03:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 409877504 bytes 2024-03-11T23:03:57.886 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 486099968 bytes 2024-03-11T23:03:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 571037696 bytes 2024-03-11T23:03:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 605598720 bytes 2024-03-11T23:03:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 658642944 bytes 2024-03-11T23:03:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 692593664 bytes 2024-03-11T23:03:58.919 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 733227008 bytes 2024-03-11T23:03:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 761788416 bytes 2024-03-11T23:03:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 797132800 bytes 2024-03-11T23:03:59.398 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 816205824 bytes 2024-03-11T23:03:59.571 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 838299648 bytes 2024-03-11T23:03:59.710 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 853352448 bytes 2024-03-11T23:03:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 913345536 bytes 2024-03-11T23:04:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 948317184 bytes 2024-03-11T23:04:00.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 1034639360 bytes 2024-03-11T23:04:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 1046367232 bytes 2024-03-11T23:04:00.591 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 1057534976 bytes 2024-03-11T23:04:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1082719232 bytes 2024-03-11T23:04:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1127328768 bytes 2024-03-11T23:04:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1193369600 bytes 2024-03-11T23:04:01.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1214702592 bytes 2024-03-11T23:04:01.446 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1247582208 bytes 2024-03-11T23:04:01.655 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1291334656 bytes 2024-03-11T23:04:01.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1314416640 bytes 2024-03-11T23:04:01.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1315030016 bytes 2024-03-11T23:04:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1395043328 bytes 2024-03-11T23:04:02.384 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1471948800 bytes 2024-03-11T23:04:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1522234368 bytes 2024-03-11T23:04:02.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1557066752 bytes 2024-03-11T23:04:02.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1598415872 bytes 2024-03-11T23:04:03.146 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1645561856 bytes 2024-03-11T23:04:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1669168128 bytes 2024-03-11T23:04:03.562 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1748533248 bytes 2024-03-11T23:04:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1784129536 bytes 2024-03-11T23:04:03.887 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1820442624 bytes 2024-03-11T23:04:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1854280704 bytes 2024-03-11T23:04:04.274 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 1913422848 bytes 2024-03-11T23:04:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 1930808320 bytes 2024-03-11T23:04:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 1982118912 bytes 2024-03-11T23:04:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 2028252160 bytes 2024-03-11T23:04:04.931 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 2051094528 bytes 2024-03-11T23:04:05.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 2128748544 bytes 2024-03-11T23:04:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 2137240576 bytes 2024-03-11T23:04:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5100 IOs, 2176718848 bytes 2024-03-11T23:04:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5200 IOs, 2251319296 bytes 2024-03-11T23:04:05.860 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5300 IOs, 2268667904 bytes 2024-03-11T23:04:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5400 IOs, 2282407936 bytes 2024-03-11T23:04:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5500 IOs, 2294709248 bytes 2024-03-11T23:04:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5600 IOs, 2341151744 bytes 2024-03-11T23:04:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5700 IOs, 2368826368 bytes 2024-03-11T23:04:06.575 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5800 IOs, 2395145216 bytes 2024-03-11T23:04:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5900 IOs, 2411780096 bytes 2024-03-11T23:04:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6000 IOs, 2417665024 bytes 2024-03-11T23:04:06.873 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:04:06.873 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:04:06.873 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:04:06.873 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:04:06.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:04:06.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_29 2024-03-11T23:04:06.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 snapshot 128M 2024-03-11T23:04:06.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:04:06.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:06.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:04:06.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:06.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_29 2024-03-11T23:04:06.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:06.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:04:06.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:04:06.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:06.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_29 128M 2024-03-11T23:04:06.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:04:06.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:06.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:04:06.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:06.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_29 2024-03-11T23:04:06.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:06.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:04:06.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:04:06.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:04:06.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:06.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_29 2024-03-11T23:04:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:04:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:04:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:04:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:04:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_29 snapshot 2024-03-11T23:04:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:04:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:04:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_29 2024-03-11T23:04:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:04:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_29 snapshot 2024-03-11T23:04:07.867 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:04:07.873 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_29 2024-03-11T23:04:07.911 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_29': 2024-03-11T23:04:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:04:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:04:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:04:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 137451b527b7 2024-03-11T23:04:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.137451b527b7 2024-03-11T23:04:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:04:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:04:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:04:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:04:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:04:06 2024 2024-03-11T23:04:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:04:06 2024 2024-03-11T23:04:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:04:06 2024 2024-03-11T23:04:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:04:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:04:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 56839585-8992-435c-80a3-a4d4b2cb652c 2024-03-11T23:04:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:04:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:04:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_29 2024-03-11T23:04:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:04:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:04:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_29 2024-03-11T23:04:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:04:07.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=7 2024-03-11T23:04:07.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:04:07.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:04:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 3597312 bytes 2024-03-11T23:04:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 45447168 bytes 2024-03-11T23:04:08.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 70925312 bytes 2024-03-11T23:04:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 92036096 bytes 2024-03-11T23:04:08.662 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 125076480 bytes 2024-03-11T23:04:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 156965888 bytes 2024-03-11T23:04:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 203810816 bytes 2024-03-11T23:04:09.260 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 279681024 bytes 2024-03-11T23:04:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 296198144 bytes 2024-03-11T23:04:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 349002752 bytes 2024-03-11T23:04:09.747 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 374729728 bytes 2024-03-11T23:04:09.910 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 410692608 bytes 2024-03-11T23:04:10.026 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 445049856 bytes 2024-03-11T23:04:10.402 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 561866752 bytes 2024-03-11T23:04:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 601470976 bytes 2024-03-11T23:04:10.722 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 631881728 bytes 2024-03-11T23:04:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 687015936 bytes 2024-03-11T23:04:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 760708096 bytes 2024-03-11T23:04:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 761338880 bytes 2024-03-11T23:04:11.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 825203712 bytes 2024-03-11T23:04:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 889860096 bytes 2024-03-11T23:04:11.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 938993664 bytes 2024-03-11T23:04:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 1001841664 bytes 2024-03-11T23:04:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 1017908224 bytes 2024-03-11T23:04:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1060894720 bytes 2024-03-11T23:04:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1076455424 bytes 2024-03-11T23:04:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1109587968 bytes 2024-03-11T23:04:13.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1164183552 bytes 2024-03-11T23:04:13.310 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1275654144 bytes 2024-03-11T23:04:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1343058944 bytes 2024-03-11T23:04:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1419882496 bytes 2024-03-11T23:04:13.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1459812352 bytes 2024-03-11T23:04:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1479223296 bytes 2024-03-11T23:04:14.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1517001728 bytes 2024-03-11T23:04:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1575524352 bytes 2024-03-11T23:04:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1630663680 bytes 2024-03-11T23:04:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:04:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:04:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:04:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:04:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:04:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_30 2024-03-11T23:04:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 snapshot 128M 2024-03-11T23:04:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:04:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:04:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_30 2024-03-11T23:04:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:04:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:04:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_30 128M 2024-03-11T23:04:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:04:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:04:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_30 2024-03-11T23:04:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:04:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:04:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:04:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_30 2024-03-11T23:04:14.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:04:14.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:04:14.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:04:14.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:04:14.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_30 snapshot 2024-03-11T23:04:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:04:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:04:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_30 2024-03-11T23:04:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:04:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_30 snapshot 2024-03-11T23:04:15.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:04:15.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_30 2024-03-11T23:04:15.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_30': 2024-03-11T23:04:15.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:04:15.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:04:15.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:04:15.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 137db192e60c 2024-03-11T23:04:15.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.137db192e60c 2024-03-11T23:04:15.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:04:15.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:04:15.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:04:15.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:04:15.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:04:14 2024 2024-03-11T23:04:15.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:04:14 2024 2024-03-11T23:04:15.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:04:14 2024 2024-03-11T23:04:15.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:04:15.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:04:15.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 5b5325b7-4350-47ba-9791-39702d0ec94a 2024-03-11T23:04:15.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:04:15.972 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:04:15.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_30 2024-03-11T23:04:15.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:04:15.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:04:15.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_30 2024-03-11T23:04:15.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:04:15.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=8 2024-03-11T23:04:15.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:04:15.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:04:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 14923776 bytes 2024-03-11T23:04:16.248 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 20257792 bytes 2024-03-11T23:04:16.461 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 77761536 bytes 2024-03-11T23:04:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 86787072 bytes 2024-03-11T23:04:16.776 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 138094592 bytes 2024-03-11T23:04:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 202528768 bytes 2024-03-11T23:04:17.257 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 299265024 bytes 2024-03-11T23:04:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 377021440 bytes 2024-03-11T23:04:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 404828160 bytes 2024-03-11T23:04:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 408138752 bytes 2024-03-11T23:04:18.017 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 462816256 bytes 2024-03-11T23:04:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 508156928 bytes 2024-03-11T23:04:18.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 508807168 bytes 2024-03-11T23:04:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 532826112 bytes 2024-03-11T23:04:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 579231744 bytes 2024-03-11T23:04:18.760 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 602036224 bytes 2024-03-11T23:04:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 658127872 bytes 2024-03-11T23:04:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 727119872 bytes 2024-03-11T23:04:19.430 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 784314368 bytes 2024-03-11T23:04:19.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 792695808 bytes 2024-03-11T23:04:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 817103872 bytes 2024-03-11T23:04:19.834 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 840425472 bytes 2024-03-11T23:04:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 914123776 bytes 2024-03-11T23:04:20.230 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 958261248 bytes 2024-03-11T23:04:20.535 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1052247040 bytes 2024-03-11T23:04:20.793 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-03-11T23:04:20.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-03-11T23:04:20.794 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-03-11T23:04:20.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr:2024-03-11T23:04:20.790+0000 7f67a59ef640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 48078) UID: 1000 2024-03-11T23:04:20.812 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1141328896 bytes 2024-03-11T23:04:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1160516608 bytes 2024-03-11T23:04:21.210 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1236114432 bytes 2024-03-11T23:04:21.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1282073600 bytes 2024-03-11T23:04:21.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1364816896 bytes 2024-03-11T23:04:21.795 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1388960768 bytes 2024-03-11T23:04:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1416854528 bytes 2024-03-11T23:04:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1452920832 bytes 2024-03-11T23:04:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1509978112 bytes 2024-03-11T23:04:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1534123008 bytes 2024-03-11T23:04:22.737 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1629533184 bytes 2024-03-11T23:04:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1654719488 bytes 2024-03-11T23:04:23.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1705128960 bytes 2024-03-11T23:04:23.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1741618176 bytes 2024-03-11T23:04:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1760870400 bytes 2024-03-11T23:04:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1807852544 bytes 2024-03-11T23:04:23.742 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1852767232 bytes 2024-03-11T23:04:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1891856384 bytes 2024-03-11T23:04:23.980 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:04:23.980 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:04:23.980 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:04:23.980 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:04:23.980 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:04:23.980 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_31 2024-03-11T23:04:23.980 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 snapshot 128M 2024-03-11T23:04:23.980 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:04:23.980 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:23.981 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:04:23.981 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:23.981 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_31 2024-03-11T23:04:23.981 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:23.981 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:04:23.981 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:04:23.981 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:23.981 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_31 128M 2024-03-11T23:04:23.981 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:04:23.981 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:23.981 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:04:23.981 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:23.981 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_31 2024-03-11T23:04:23.981 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:23.981 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:04:23.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:04:23.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:04:23.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:23.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_31 2024-03-11T23:04:24.028 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:04:24.028 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:04:24.028 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:04:24.028 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:04:24.029 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_31 snapshot 2024-03-11T23:04:24.029 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:04:24.029 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:04:24.029 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_31 2024-03-11T23:04:24.029 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:04:24.029 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_31 snapshot 2024-03-11T23:04:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:04:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_31 2024-03-11T23:04:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_31': 2024-03-11T23:04:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:04:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:04:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:04:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 13856c61405 2024-03-11T23:04:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.13856c61405 2024-03-11T23:04:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:04:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:04:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:04:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:04:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:04:24 2024 2024-03-11T23:04:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:04:24 2024 2024-03-11T23:04:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:04:24 2024 2024-03-11T23:04:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:04:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:04:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 0fbab2fa-8751-419f-a846-c51d8786c5c5 2024-03-11T23:04:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:04:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:04:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_31 2024-03-11T23:04:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:04:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:04:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_31 2024-03-11T23:04:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:04:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=5 2024-03-11T23:04:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:04:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:04:25.184 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 21430272 bytes 2024-03-11T23:04:25.402 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 98176000 bytes 2024-03-11T23:04:25.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 134278144 bytes 2024-03-11T23:04:25.752 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 170200064 bytes 2024-03-11T23:04:25.795 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-03-11T23:04:25.795 INFO:teuthology.orchestra.run:waiting for 600 2024-03-11T23:04:25.795 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-03-11T23:04:25.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-03-11T23:04:25.796 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-03-11T23:04:25.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-03-11T23:04:25.798 INFO:tasks.rbd_mirror_thrash:waiting for 110 secs before thrashing 2024-03-11T23:04:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 223876096 bytes 2024-03-11T23:04:26.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 245642240 bytes 2024-03-11T23:04:26.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 307219456 bytes 2024-03-11T23:04:26.584 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 358330368 bytes 2024-03-11T23:04:26.735 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 386348032 bytes 2024-03-11T23:04:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 462724096 bytes 2024-03-11T23:04:27.113 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 481925120 bytes 2024-03-11T23:04:27.306 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 528894976 bytes 2024-03-11T23:04:27.422 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 535207936 bytes 2024-03-11T23:04:27.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 568086528 bytes 2024-03-11T23:04:27.779 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 611899392 bytes 2024-03-11T23:04:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 675750912 bytes 2024-03-11T23:04:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 726507520 bytes 2024-03-11T23:04:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 745905152 bytes 2024-03-11T23:04:28.521 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 805705728 bytes 2024-03-11T23:04:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 908937216 bytes 2024-03-11T23:04:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 931255296 bytes 2024-03-11T23:04:29.173 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 988399616 bytes 2024-03-11T23:04:29.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 1027577856 bytes 2024-03-11T23:04:29.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 1083025408 bytes 2024-03-11T23:04:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1170929664 bytes 2024-03-11T23:04:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:04:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:04:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:04:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:04:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:04:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_32 2024-03-11T23:04:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 snapshot 128M 2024-03-11T23:04:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:04:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:04:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_32 2024-03-11T23:04:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:04:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:04:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_32 128M 2024-03-11T23:04:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:04:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:04:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_32 2024-03-11T23:04:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:04:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:04:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:04:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_32 2024-03-11T23:04:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:04:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:04:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:04:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:04:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_32 snapshot 2024-03-11T23:04:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:04:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:04:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_32 2024-03-11T23:04:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:04:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_32 snapshot 2024-03-11T23:04:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:04:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_32 2024-03-11T23:04:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_32': 2024-03-11T23:04:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:04:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:04:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:04:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 1397f283d1be 2024-03-11T23:04:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.1397f283d1be 2024-03-11T23:04:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:04:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:04:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:04:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:04:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:04:30 2024 2024-03-11T23:04:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:04:30 2024 2024-03-11T23:04:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:04:30 2024 2024-03-11T23:04:31.058 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:04:31.058 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:04:31.058 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: adeee68d-39bd-4c47-9b90-d83d5ac98497 2024-03-11T23:04:31.058 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:04:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:04:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_32 2024-03-11T23:04:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:04:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:04:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_32 2024-03-11T23:04:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:04:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=9 2024-03-11T23:04:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:04:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:04:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 40238080 bytes 2024-03-11T23:04:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 100678656 bytes 2024-03-11T23:04:31.752 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 189496320 bytes 2024-03-11T23:04:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 237690880 bytes 2024-03-11T23:04:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 286341120 bytes 2024-03-11T23:04:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 308359168 bytes 2024-03-11T23:04:32.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 384403456 bytes 2024-03-11T23:04:32.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 455107584 bytes 2024-03-11T23:04:32.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 460172288 bytes 2024-03-11T23:04:33.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 460790784 bytes 2024-03-11T23:04:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 499290112 bytes 2024-03-11T23:04:33.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 536870912 bytes 2024-03-11T23:04:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 574412800 bytes 2024-03-11T23:04:33.800 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 656350208 bytes 2024-03-11T23:04:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 677131264 bytes 2024-03-11T23:04:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 695470080 bytes 2024-03-11T23:04:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 720388096 bytes 2024-03-11T23:04:34.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 738658304 bytes 2024-03-11T23:04:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 791473152 bytes 2024-03-11T23:04:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 831009792 bytes 2024-03-11T23:04:34.837 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 831621120 bytes 2024-03-11T23:04:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 881051648 bytes 2024-03-11T23:04:35.211 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 903188480 bytes 2024-03-11T23:04:35.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 936238080 bytes 2024-03-11T23:04:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 943717376 bytes 2024-03-11T23:04:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1015945216 bytes 2024-03-11T23:04:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1099480064 bytes 2024-03-11T23:04:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1126955008 bytes 2024-03-11T23:04:36.352 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1171984384 bytes 2024-03-11T23:04:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1203444736 bytes 2024-03-11T23:04:36.707 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1250572288 bytes 2024-03-11T23:04:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1287052288 bytes 2024-03-11T23:04:37.114 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1363263488 bytes 2024-03-11T23:04:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1388127232 bytes 2024-03-11T23:04:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1399725056 bytes 2024-03-11T23:04:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1422962688 bytes 2024-03-11T23:04:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1468187648 bytes 2024-03-11T23:04:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1541321728 bytes 2024-03-11T23:04:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1613584384 bytes 2024-03-11T23:04:38.388 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1655964672 bytes 2024-03-11T23:04:38.613 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1716136960 bytes 2024-03-11T23:04:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1716780032 bytes 2024-03-11T23:04:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1774643200 bytes 2024-03-11T23:04:39.131 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 1822838784 bytes 2024-03-11T23:04:39.301 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 1864927232 bytes 2024-03-11T23:04:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 1886431232 bytes 2024-03-11T23:04:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 1887035392 bytes 2024-03-11T23:04:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 1956746240 bytes 2024-03-11T23:04:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:04:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:04:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:04:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:04:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:04:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_33 2024-03-11T23:04:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 snapshot 128M 2024-03-11T23:04:40.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:04:40.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:40.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:04:40.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:40.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_33 2024-03-11T23:04:40.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:40.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:04:40.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:04:40.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:40.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_33 128M 2024-03-11T23:04:40.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:04:40.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:40.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:04:40.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:40.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_33 2024-03-11T23:04:40.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:40.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:04:40.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:04:40.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:04:40.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:40.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_33 2024-03-11T23:04:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:04:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:04:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:04:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:04:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_33 snapshot 2024-03-11T23:04:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:04:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:04:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_33 2024-03-11T23:04:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:04:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_33 snapshot 2024-03-11T23:04:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:04:41.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_33 2024-03-11T23:04:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_33': 2024-03-11T23:04:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:04:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:04:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:04:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 13a07d8cae45 2024-03-11T23:04:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.13a07d8cae45 2024-03-11T23:04:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:04:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:04:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:04:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:04:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:04:40 2024 2024-03-11T23:04:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:04:40 2024 2024-03-11T23:04:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:04:40 2024 2024-03-11T23:04:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:04:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:04:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 94a5319a-2e74-4f65-a321-3f962cd4ebe5 2024-03-11T23:04:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:04:41.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:04:41.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_33 2024-03-11T23:04:41.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:04:41.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:04:41.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_33 2024-03-11T23:04:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:04:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=5 2024-03-11T23:04:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:04:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:04:41.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 54234112 bytes 2024-03-11T23:04:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 124785664 bytes 2024-03-11T23:04:41.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 147137536 bytes 2024-03-11T23:04:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 198788096 bytes 2024-03-11T23:04:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 199412736 bytes 2024-03-11T23:04:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 210878464 bytes 2024-03-11T23:04:42.303 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 238084096 bytes 2024-03-11T23:04:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 315785216 bytes 2024-03-11T23:04:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 411662336 bytes 2024-03-11T23:04:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 463268864 bytes 2024-03-11T23:04:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 535491584 bytes 2024-03-11T23:04:43.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 536101888 bytes 2024-03-11T23:04:43.566 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 560409600 bytes 2024-03-11T23:04:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 620616704 bytes 2024-03-11T23:04:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 621239296 bytes 2024-03-11T23:04:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 649676800 bytes 2024-03-11T23:04:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 663934976 bytes 2024-03-11T23:04:44.424 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 770583552 bytes 2024-03-11T23:04:44.597 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 788467712 bytes 2024-03-11T23:04:44.790 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 833624064 bytes 2024-03-11T23:04:45.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 891939840 bytes 2024-03-11T23:04:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 926448640 bytes 2024-03-11T23:04:45.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 957981696 bytes 2024-03-11T23:04:45.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 1024441344 bytes 2024-03-11T23:04:45.683 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1038408704 bytes 2024-03-11T23:04:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1069765632 bytes 2024-03-11T23:04:46.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1126014976 bytes 2024-03-11T23:04:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:04:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:04:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:04:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:04:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:04:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_34 2024-03-11T23:04:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 snapshot 128M 2024-03-11T23:04:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:04:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:04:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_34 2024-03-11T23:04:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:04:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:04:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_34 128M 2024-03-11T23:04:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:04:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:04:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_34 2024-03-11T23:04:46.150 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:46.150 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:04:46.150 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:04:46.150 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:04:46.150 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:46.150 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_34 2024-03-11T23:04:46.193 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:04:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:04:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:04:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:04:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_34 snapshot 2024-03-11T23:04:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:04:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:04:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_34 2024-03-11T23:04:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:04:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_34 snapshot 2024-03-11T23:04:46.303 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:04:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_34 2024-03-11T23:04:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_34': 2024-03-11T23:04:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:04:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:04:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:04:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 13aa326cc233 2024-03-11T23:04:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.13aa326cc233 2024-03-11T23:04:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:04:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:04:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:04:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:04:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:04:46 2024 2024-03-11T23:04:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:04:46 2024 2024-03-11T23:04:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:04:46 2024 2024-03-11T23:04:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:04:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:04:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: fa15ec82-36d2-421d-b90c-12034ab39d68 2024-03-11T23:04:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:04:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:04:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_34 2024-03-11T23:04:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:04:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:04:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_34 2024-03-11T23:04:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:04:46.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=11 2024-03-11T23:04:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:04:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:04:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 101349376 bytes 2024-03-11T23:04:46.793 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 101970944 bytes 2024-03-11T23:04:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 174114816 bytes 2024-03-11T23:04:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 298042368 bytes 2024-03-11T23:04:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 355446784 bytes 2024-03-11T23:04:47.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 404982784 bytes 2024-03-11T23:04:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 430499840 bytes 2024-03-11T23:04:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 525725696 bytes 2024-03-11T23:04:48.404 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 580462592 bytes 2024-03-11T23:04:48.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 644390912 bytes 2024-03-11T23:04:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 648333312 bytes 2024-03-11T23:04:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 701920256 bytes 2024-03-11T23:04:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 747994112 bytes 2024-03-11T23:04:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 791715840 bytes 2024-03-11T23:04:49.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 869612544 bytes 2024-03-11T23:04:49.782 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 921696256 bytes 2024-03-11T23:04:50.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 988302336 bytes 2024-03-11T23:04:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 1028452352 bytes 2024-03-11T23:04:50.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 1041007616 bytes 2024-03-11T23:04:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 1107136512 bytes 2024-03-11T23:04:50.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 1129817088 bytes 2024-03-11T23:04:51.008 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 1230385152 bytes 2024-03-11T23:04:51.189 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 1270306816 bytes 2024-03-11T23:04:51.335 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 1287933952 bytes 2024-03-11T23:04:51.556 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1351184384 bytes 2024-03-11T23:04:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1374895104 bytes 2024-03-11T23:04:51.878 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1410958336 bytes 2024-03-11T23:04:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1489983488 bytes 2024-03-11T23:04:52.322 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1533264896 bytes 2024-03-11T23:04:52.424 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1557592064 bytes 2024-03-11T23:04:52.651 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1610553344 bytes 2024-03-11T23:04:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1622548480 bytes 2024-03-11T23:04:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1663043584 bytes 2024-03-11T23:04:53.080 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1677680640 bytes 2024-03-11T23:04:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1755964416 bytes 2024-03-11T23:04:53.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1764113408 bytes 2024-03-11T23:04:53.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1791357952 bytes 2024-03-11T23:04:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1870086144 bytes 2024-03-11T23:04:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1893327872 bytes 2024-03-11T23:04:54.275 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1900033024 bytes 2024-03-11T23:04:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1959799808 bytes 2024-03-11T23:04:54.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 2016630784 bytes 2024-03-11T23:04:54.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 2110980096 bytes 2024-03-11T23:04:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 2148981760 bytes 2024-03-11T23:04:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 2251816960 bytes 2024-03-11T23:04:55.748 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 2334332928 bytes 2024-03-11T23:04:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 2378021888 bytes 2024-03-11T23:04:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 2383188992 bytes 2024-03-11T23:04:56.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 2490990592 bytes 2024-03-11T23:04:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 2531252224 bytes 2024-03-11T23:04:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5100 IOs, 2589785088 bytes 2024-03-11T23:04:56.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5200 IOs, 2616640512 bytes 2024-03-11T23:04:57.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5300 IOs, 2637335552 bytes 2024-03-11T23:04:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5400 IOs, 2706935808 bytes 2024-03-11T23:04:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:04:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:04:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:04:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:04:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:04:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_35 2024-03-11T23:04:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 snapshot 128M 2024-03-11T23:04:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:04:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:04:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_35 2024-03-11T23:04:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:04:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:04:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_35 128M 2024-03-11T23:04:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:04:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:04:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_35 2024-03-11T23:04:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:04:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:04:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:04:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:04:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_35 2024-03-11T23:04:57.401 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:04:57.401 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:04:57.401 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:04:57.401 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:04:57.401 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_35 snapshot 2024-03-11T23:04:57.401 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:04:57.401 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:04:57.401 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_35 2024-03-11T23:04:57.402 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:04:57.402 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_35 snapshot 2024-03-11T23:04:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:04:57.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_35 2024-03-11T23:04:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_35': 2024-03-11T23:04:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:04:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:04:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:04:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 13b26ba2dfe7 2024-03-11T23:04:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.13b26ba2dfe7 2024-03-11T23:04:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:04:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:04:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:04:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:04:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:04:57 2024 2024-03-11T23:04:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:04:57 2024 2024-03-11T23:04:57.980 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:04:57 2024 2024-03-11T23:04:57.980 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:04:57.980 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:04:57.980 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: ddb7a161-de34-428e-ae0c-47939be794d1 2024-03-11T23:04:57.980 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:04:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:04:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_35 2024-03-11T23:04:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:04:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:04:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_35 2024-03-11T23:04:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:04:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=10 2024-03-11T23:04:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:04:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:04:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 54409216 bytes 2024-03-11T23:04:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 119591936 bytes 2024-03-11T23:04:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 136770560 bytes 2024-03-11T23:04:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 192387072 bytes 2024-03-11T23:04:58.901 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 202370048 bytes 2024-03-11T23:04:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 287835136 bytes 2024-03-11T23:04:59.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 342951936 bytes 2024-03-11T23:04:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 371387392 bytes 2024-03-11T23:04:59.724 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 434639872 bytes 2024-03-11T23:04:59.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 442992640 bytes 2024-03-11T23:05:00.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 494644224 bytes 2024-03-11T23:05:00.189 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 521460736 bytes 2024-03-11T23:05:00.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 584409088 bytes 2024-03-11T23:05:00.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 616329216 bytes 2024-03-11T23:05:00.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 647422976 bytes 2024-03-11T23:05:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 671922176 bytes 2024-03-11T23:05:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 718235648 bytes 2024-03-11T23:05:01.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 799601664 bytes 2024-03-11T23:05:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 836814848 bytes 2024-03-11T23:05:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 929061888 bytes 2024-03-11T23:05:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 962850816 bytes 2024-03-11T23:05:02.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 1050856448 bytes 2024-03-11T23:05:02.570 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 1149847552 bytes 2024-03-11T23:05:02.808 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 1200679936 bytes 2024-03-11T23:05:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1250618368 bytes 2024-03-11T23:05:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1268646912 bytes 2024-03-11T23:05:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1303034880 bytes 2024-03-11T23:05:03.520 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1347022848 bytes 2024-03-11T23:05:03.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1371385856 bytes 2024-03-11T23:05:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1445512192 bytes 2024-03-11T23:05:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1468504064 bytes 2024-03-11T23:05:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1489090560 bytes 2024-03-11T23:05:04.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1513879552 bytes 2024-03-11T23:05:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1542866944 bytes 2024-03-11T23:05:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1583079424 bytes 2024-03-11T23:05:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1624730624 bytes 2024-03-11T23:05:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1647802368 bytes 2024-03-11T23:05:05.134 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1651027968 bytes 2024-03-11T23:05:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1665708032 bytes 2024-03-11T23:05:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1677210624 bytes 2024-03-11T23:05:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1694136320 bytes 2024-03-11T23:05:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1697172480 bytes 2024-03-11T23:05:05.697 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9326978543999421) > weight (0.25) 2024-03-11T23:05:05.698 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5272716426690501) > weight (0.25) 2024-03-11T23:05:05.698 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-03-11T23:05:05.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-03-11T23:05:05.698 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-03-11T23:05:05.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi132.stderr:2024-03-11T23:05:05.695+0000 7fbbe80a6640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 47460) UID: 1000 2024-03-11T23:05:05.778 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1720254464 bytes 2024-03-11T23:05:05.933 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 1745553408 bytes 2024-03-11T23:05:06.135 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 1798720512 bytes 2024-03-11T23:05:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 1826840576 bytes 2024-03-11T23:05:06.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 1871958016 bytes 2024-03-11T23:05:06.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 1883942912 bytes 2024-03-11T23:05:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 1948427264 bytes 2024-03-11T23:05:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 2001823744 bytes 2024-03-11T23:05:07.259 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5100 IOs, 2066475008 bytes 2024-03-11T23:05:07.632 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5200 IOs, 2206282752 bytes 2024-03-11T23:05:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5300 IOs, 2228122624 bytes 2024-03-11T23:05:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5400 IOs, 2239466496 bytes 2024-03-11T23:05:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:05:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:05:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:05:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:05:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:05:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_36 2024-03-11T23:05:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 snapshot 128M 2024-03-11T23:05:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:05:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:05:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_36 2024-03-11T23:05:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:05:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:05:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_36 128M 2024-03-11T23:05:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:05:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:05:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_36 2024-03-11T23:05:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:05:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:05:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:05:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_36 2024-03-11T23:05:08.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:05:08.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:05:08.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:05:08.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:05:08.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_36 snapshot 2024-03-11T23:05:08.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:05:08.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:05:08.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_36 2024-03-11T23:05:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:05:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_36 snapshot 2024-03-11T23:05:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:05:08.960 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_36 2024-03-11T23:05:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_36': 2024-03-11T23:05:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:05:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:05:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:05:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 13bb516d0a04 2024-03-11T23:05:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.13bb516d0a04 2024-03-11T23:05:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:05:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:05:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:05:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:05:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:05:08 2024 2024-03-11T23:05:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:05:08 2024 2024-03-11T23:05:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:05:08 2024 2024-03-11T23:05:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:05:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:05:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 0ecf1da7-76ab-4437-ad6e-4ee82f47a299 2024-03-11T23:05:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:05:08.999 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:05:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_36 2024-03-11T23:05:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:05:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:05:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_36 2024-03-11T23:05:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:05:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=8 2024-03-11T23:05:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:05:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:05:09.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 47005696 bytes 2024-03-11T23:05:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 78698496 bytes 2024-03-11T23:05:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 124815360 bytes 2024-03-11T23:05:09.745 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 166175744 bytes 2024-03-11T23:05:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 188136448 bytes 2024-03-11T23:05:10.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 232414208 bytes 2024-03-11T23:05:10.298 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 288671744 bytes 2024-03-11T23:05:10.478 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 334488576 bytes 2024-03-11T23:05:10.679 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 380153856 bytes 2024-03-11T23:05:10.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 422791168 bytes 2024-03-11T23:05:11.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 439333888 bytes 2024-03-11T23:05:11.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 477413376 bytes 2024-03-11T23:05:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 570969088 bytes 2024-03-11T23:05:11.699 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-03-11T23:05:11.699 INFO:teuthology.orchestra.run:waiting for 600 2024-03-11T23:05:11.699 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-03-11T23:05:11.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-03-11T23:05:11.700 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-03-11T23:05:11.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-03-11T23:05:11.704 INFO:tasks.rbd_mirror_thrash:waiting for 66 secs before thrashing 2024-03-11T23:05:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 637348864 bytes 2024-03-11T23:05:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 746426368 bytes 2024-03-11T23:05:12.130 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 747036672 bytes 2024-03-11T23:05:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 795461632 bytes 2024-03-11T23:05:12.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 839570432 bytes 2024-03-11T23:05:12.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 885725184 bytes 2024-03-11T23:05:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 928158720 bytes 2024-03-11T23:05:13.047 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 952640512 bytes 2024-03-11T23:05:13.244 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 1017027584 bytes 2024-03-11T23:05:13.415 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 1042506752 bytes 2024-03-11T23:05:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 1064543232 bytes 2024-03-11T23:05:13.788 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1124671488 bytes 2024-03-11T23:05:13.906 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1131490304 bytes 2024-03-11T23:05:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1136713728 bytes 2024-03-11T23:05:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1169955840 bytes 2024-03-11T23:05:14.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1197705216 bytes 2024-03-11T23:05:14.573 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1266849792 bytes 2024-03-11T23:05:14.681 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1270377472 bytes 2024-03-11T23:05:14.896 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1321578496 bytes 2024-03-11T23:05:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1322177536 bytes 2024-03-11T23:05:15.145 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1367925760 bytes 2024-03-11T23:05:15.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1445355520 bytes 2024-03-11T23:05:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1529958400 bytes 2024-03-11T23:05:15.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1559467008 bytes 2024-03-11T23:05:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1560093696 bytes 2024-03-11T23:05:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1591145472 bytes 2024-03-11T23:05:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1717483520 bytes 2024-03-11T23:05:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1733328896 bytes 2024-03-11T23:05:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1763310592 bytes 2024-03-11T23:05:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1777389568 bytes 2024-03-11T23:05:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:05:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:05:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:05:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:05:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:05:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_37 2024-03-11T23:05:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 snapshot 128M 2024-03-11T23:05:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:05:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:05:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_37 2024-03-11T23:05:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:05:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:05:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_37 128M 2024-03-11T23:05:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:05:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:05:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_37 2024-03-11T23:05:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:05:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:05:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:05:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_37 2024-03-11T23:05:17.253 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:05:17.253 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:05:17.253 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:05:17.253 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:05:17.254 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_37 snapshot 2024-03-11T23:05:17.254 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:05:17.254 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:05:17.254 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_37 2024-03-11T23:05:17.254 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:05:17.254 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_37 snapshot 2024-03-11T23:05:18.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:05:18.015 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_37 2024-03-11T23:05:18.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_37': 2024-03-11T23:05:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:05:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:05:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:05:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 13cda2287ca4 2024-03-11T23:05:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.13cda2287ca4 2024-03-11T23:05:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:05:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:05:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:05:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:05:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:05:17 2024 2024-03-11T23:05:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:05:17 2024 2024-03-11T23:05:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:05:17 2024 2024-03-11T23:05:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:05:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:05:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 9e66f1a3-2f01-4cca-827d-d5d0c892aa22 2024-03-11T23:05:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:05:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:05:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_37 2024-03-11T23:05:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:05:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:05:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_37 2024-03-11T23:05:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:05:18.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=6 2024-03-11T23:05:18.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:05:18.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:05:18.316 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 58371072 bytes 2024-03-11T23:05:18.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 124060672 bytes 2024-03-11T23:05:18.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 136407040 bytes 2024-03-11T23:05:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 209913856 bytes 2024-03-11T23:05:19.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 210611200 bytes 2024-03-11T23:05:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 261280768 bytes 2024-03-11T23:05:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 304556032 bytes 2024-03-11T23:05:19.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 355273728 bytes 2024-03-11T23:05:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 404677632 bytes 2024-03-11T23:05:19.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 427849728 bytes 2024-03-11T23:05:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 436398080 bytes 2024-03-11T23:05:20.311 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 513979392 bytes 2024-03-11T23:05:20.452 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 528212992 bytes 2024-03-11T23:05:20.681 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 598044672 bytes 2024-03-11T23:05:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 673372160 bytes 2024-03-11T23:05:21.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 680811520 bytes 2024-03-11T23:05:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 718613504 bytes 2024-03-11T23:05:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 746548224 bytes 2024-03-11T23:05:21.572 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 810243072 bytes 2024-03-11T23:05:21.721 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 821767168 bytes 2024-03-11T23:05:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 835974144 bytes 2024-03-11T23:05:21.996 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 866963456 bytes 2024-03-11T23:05:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 947849216 bytes 2024-03-11T23:05:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 990683136 bytes 2024-03-11T23:05:22.562 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1000010752 bytes 2024-03-11T23:05:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1053122560 bytes 2024-03-11T23:05:22.967 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1118346240 bytes 2024-03-11T23:05:23.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1134382080 bytes 2024-03-11T23:05:23.217 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1149775872 bytes 2024-03-11T23:05:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1221904384 bytes 2024-03-11T23:05:23.622 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1242095616 bytes 2024-03-11T23:05:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1279114240 bytes 2024-03-11T23:05:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1333999616 bytes 2024-03-11T23:05:24.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:05:24.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:05:24.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:05:24.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:05:24.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:05:24.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_38 2024-03-11T23:05:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 snapshot 128M 2024-03-11T23:05:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:05:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:05:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_38 2024-03-11T23:05:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:05:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:05:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_38 128M 2024-03-11T23:05:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:05:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:05:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_38 2024-03-11T23:05:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:05:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:05:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:05:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_38 2024-03-11T23:05:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:05:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:05:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:05:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:05:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_38 snapshot 2024-03-11T23:05:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:05:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:05:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_38 2024-03-11T23:05:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:05:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_38 snapshot 2024-03-11T23:05:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:05:25.029 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_38 2024-03-11T23:05:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_38': 2024-03-11T23:05:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:05:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:05:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:05:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 13d95e0733bd 2024-03-11T23:05:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.13d95e0733bd 2024-03-11T23:05:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:05:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:05:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:05:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:05:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:05:24 2024 2024-03-11T23:05:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:05:24 2024 2024-03-11T23:05:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:05:24 2024 2024-03-11T23:05:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:05:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:05:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 0a89ddb2-e0b9-4fa8-92a7-04ff8a761c42 2024-03-11T23:05:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:05:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:05:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_38 2024-03-11T23:05:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:05:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:05:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_38 2024-03-11T23:05:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:05:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=5 2024-03-11T23:05:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:05:25.072 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:05:25.266 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 28398592 bytes 2024-03-11T23:05:25.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 42622976 bytes 2024-03-11T23:05:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 43229184 bytes 2024-03-11T23:05:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 86508544 bytes 2024-03-11T23:05:25.805 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 114221056 bytes 2024-03-11T23:05:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 114846720 bytes 2024-03-11T23:05:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 220277760 bytes 2024-03-11T23:05:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 245766144 bytes 2024-03-11T23:05:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 246379520 bytes 2024-03-11T23:05:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 250916864 bytes 2024-03-11T23:05:26.726 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 278262784 bytes 2024-03-11T23:05:26.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 344920064 bytes 2024-03-11T23:05:27.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 389976064 bytes 2024-03-11T23:05:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 495952896 bytes 2024-03-11T23:05:27.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 567901184 bytes 2024-03-11T23:05:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 602908672 bytes 2024-03-11T23:05:28.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 696057856 bytes 2024-03-11T23:05:28.281 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 716670976 bytes 2024-03-11T23:05:28.523 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 783144960 bytes 2024-03-11T23:05:28.629 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 787399680 bytes 2024-03-11T23:05:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 826840064 bytes 2024-03-11T23:05:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 931442688 bytes 2024-03-11T23:05:29.353 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 988210176 bytes 2024-03-11T23:05:29.542 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 1037031424 bytes 2024-03-11T23:05:29.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1053159424 bytes 2024-03-11T23:05:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1079941120 bytes 2024-03-11T23:05:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1110754304 bytes 2024-03-11T23:05:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:05:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:05:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:05:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:05:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:05:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_39 2024-03-11T23:05:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 snapshot 128M 2024-03-11T23:05:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:05:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:05:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_39 2024-03-11T23:05:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:05:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:05:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_39 128M 2024-03-11T23:05:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:05:30.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:30.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:05:30.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:30.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_39 2024-03-11T23:05:30.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:30.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:05:30.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:05:30.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:05:30.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:30.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_39 2024-03-11T23:05:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:05:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:05:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:05:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:05:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_39 snapshot 2024-03-11T23:05:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:05:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:05:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_39 2024-03-11T23:05:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:05:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_39 snapshot 2024-03-11T23:05:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:05:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_39 2024-03-11T23:05:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_39': 2024-03-11T23:05:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:05:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:05:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:05:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 13e5678581ff 2024-03-11T23:05:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.13e5678581ff 2024-03-11T23:05:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:05:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:05:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:05:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:05:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:05:30 2024 2024-03-11T23:05:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:05:30 2024 2024-03-11T23:05:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:05:30 2024 2024-03-11T23:05:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:05:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:05:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: c85a3f10-e75f-4dc5-98d4-8bf7b2b4e341 2024-03-11T23:05:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:05:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:05:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_39 2024-03-11T23:05:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:05:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:05:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_39 2024-03-11T23:05:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:05:31.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=13 2024-03-11T23:05:31.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:05:31.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:05:31.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 40415232 bytes 2024-03-11T23:05:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 162659328 bytes 2024-03-11T23:05:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 183030784 bytes 2024-03-11T23:05:31.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 262315008 bytes 2024-03-11T23:05:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 306859008 bytes 2024-03-11T23:05:32.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 369611776 bytes 2024-03-11T23:05:32.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 396520448 bytes 2024-03-11T23:05:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 501118976 bytes 2024-03-11T23:05:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 501721088 bytes 2024-03-11T23:05:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 529246208 bytes 2024-03-11T23:05:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 572443648 bytes 2024-03-11T23:05:33.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 584097792 bytes 2024-03-11T23:05:33.610 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 637040640 bytes 2024-03-11T23:05:33.910 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 732907520 bytes 2024-03-11T23:05:34.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 773415936 bytes 2024-03-11T23:05:34.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 842767360 bytes 2024-03-11T23:05:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 901603328 bytes 2024-03-11T23:05:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 944352256 bytes 2024-03-11T23:05:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 1008381952 bytes 2024-03-11T23:05:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 1035821056 bytes 2024-03-11T23:05:35.400 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 1121700864 bytes 2024-03-11T23:05:35.602 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 1168782336 bytes 2024-03-11T23:05:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 1195479040 bytes 2024-03-11T23:05:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 1248868352 bytes 2024-03-11T23:05:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1282824192 bytes 2024-03-11T23:05:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1283443712 bytes 2024-03-11T23:05:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1375353856 bytes 2024-03-11T23:05:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1400882176 bytes 2024-03-11T23:05:36.786 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1438631936 bytes 2024-03-11T23:05:36.908 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1456696320 bytes 2024-03-11T23:05:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1487683584 bytes 2024-03-11T23:05:37.238 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1524743168 bytes 2024-03-11T23:05:37.516 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1600949248 bytes 2024-03-11T23:05:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1617987584 bytes 2024-03-11T23:05:37.913 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1697061888 bytes 2024-03-11T23:05:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1747725312 bytes 2024-03-11T23:05:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1769202688 bytes 2024-03-11T23:05:38.416 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1798203392 bytes 2024-03-11T23:05:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1855335424 bytes 2024-03-11T23:05:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1903589376 bytes 2024-03-11T23:05:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1907441664 bytes 2024-03-11T23:05:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1981463552 bytes 2024-03-11T23:05:39.299 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1999350784 bytes 2024-03-11T23:05:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 2047990784 bytes 2024-03-11T23:05:39.821 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 2153746432 bytes 2024-03-11T23:05:40.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 2214780928 bytes 2024-03-11T23:05:40.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 2236294144 bytes 2024-03-11T23:05:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 2247686144 bytes 2024-03-11T23:05:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 2297763840 bytes 2024-03-11T23:05:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 2362319872 bytes 2024-03-11T23:05:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5100 IOs, 2401143808 bytes 2024-03-11T23:05:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5200 IOs, 2456511488 bytes 2024-03-11T23:05:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5300 IOs, 2482161664 bytes 2024-03-11T23:05:41.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5400 IOs, 2530856960 bytes 2024-03-11T23:05:41.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5500 IOs, 2565952512 bytes 2024-03-11T23:05:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5600 IOs, 2571678720 bytes 2024-03-11T23:05:42.014 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5700 IOs, 2650509312 bytes 2024-03-11T23:05:42.210 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5800 IOs, 2699467776 bytes 2024-03-11T23:05:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5900 IOs, 2778021888 bytes 2024-03-11T23:05:42.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6000 IOs, 2822305792 bytes 2024-03-11T23:05:42.778 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6100 IOs, 2843474944 bytes 2024-03-11T23:05:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6200 IOs, 2918343680 bytes 2024-03-11T23:05:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6300 IOs, 2922370048 bytes 2024-03-11T23:05:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6400 IOs, 2995500032 bytes 2024-03-11T23:05:43.537 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6500 IOs, 3036065792 bytes 2024-03-11T23:05:43.690 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6600 IOs, 3064838144 bytes 2024-03-11T23:05:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6700 IOs, 3102855168 bytes 2024-03-11T23:05:44.074 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6800 IOs, 3156062208 bytes 2024-03-11T23:05:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:05:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:05:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:05:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:05:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:05:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_40 2024-03-11T23:05:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 snapshot 128M 2024-03-11T23:05:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:05:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:05:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_40 2024-03-11T23:05:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:05:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:05:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_40 128M 2024-03-11T23:05:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:05:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:05:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_40 2024-03-11T23:05:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:05:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:05:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:05:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_40 2024-03-11T23:05:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:05:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:05:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:05:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:05:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_40 snapshot 2024-03-11T23:05:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:05:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:05:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_40 2024-03-11T23:05:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:05:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_40 snapshot 2024-03-11T23:05:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:05:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_40 2024-03-11T23:05:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_40': 2024-03-11T23:05:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:05:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:05:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:05:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 13f15bef4795 2024-03-11T23:05:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.13f15bef4795 2024-03-11T23:05:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:05:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:05:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:05:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:05:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:05:44 2024 2024-03-11T23:05:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:05:44 2024 2024-03-11T23:05:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:05:44 2024 2024-03-11T23:05:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:05:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:05:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 5c2e35fd-9ddc-477b-abb2-86d6a62016b3 2024-03-11T23:05:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:05:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:05:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_40 2024-03-11T23:05:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:05:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:05:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_40 2024-03-11T23:05:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:05:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=7 2024-03-11T23:05:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:05:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:05:45.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 45435904 bytes 2024-03-11T23:05:45.555 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 84420608 bytes 2024-03-11T23:05:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 116914176 bytes 2024-03-11T23:05:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 182081536 bytes 2024-03-11T23:05:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 190674944 bytes 2024-03-11T23:05:46.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 209436672 bytes 2024-03-11T23:05:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 266369024 bytes 2024-03-11T23:05:46.609 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 330847232 bytes 2024-03-11T23:05:46.794 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 366606336 bytes 2024-03-11T23:05:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 418641920 bytes 2024-03-11T23:05:47.150 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 443328512 bytes 2024-03-11T23:05:47.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 506541056 bytes 2024-03-11T23:05:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 580638720 bytes 2024-03-11T23:05:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 621274112 bytes 2024-03-11T23:05:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 701528064 bytes 2024-03-11T23:05:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 776260608 bytes 2024-03-11T23:05:48.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 858170368 bytes 2024-03-11T23:05:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 912960512 bytes 2024-03-11T23:05:48.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 934998016 bytes 2024-03-11T23:05:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 968117248 bytes 2024-03-11T23:05:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 1050554368 bytes 2024-03-11T23:05:49.543 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 1093057536 bytes 2024-03-11T23:05:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 1139023872 bytes 2024-03-11T23:05:50.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 1231377408 bytes 2024-03-11T23:05:50.194 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1289153536 bytes 2024-03-11T23:05:50.483 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1345944576 bytes 2024-03-11T23:05:50.683 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1395129344 bytes 2024-03-11T23:05:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1443738624 bytes 2024-03-11T23:05:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1499634688 bytes 2024-03-11T23:05:51.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1525729280 bytes 2024-03-11T23:05:51.400 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1553617920 bytes 2024-03-11T23:05:51.535 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1573160960 bytes 2024-03-11T23:05:51.780 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1643301888 bytes 2024-03-11T23:05:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1646075904 bytes 2024-03-11T23:05:52.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1686943744 bytes 2024-03-11T23:05:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:05:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:05:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:05:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:05:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:05:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_41 2024-03-11T23:05:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 snapshot 128M 2024-03-11T23:05:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:05:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:05:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_41 2024-03-11T23:05:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:05:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:05:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_41 128M 2024-03-11T23:05:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:05:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:05:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_41 2024-03-11T23:05:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:05:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:05:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:05:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:05:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_41 2024-03-11T23:05:52.214 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:05:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:05:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:05:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:05:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_41 snapshot 2024-03-11T23:05:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:05:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:05:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_41 2024-03-11T23:05:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:05:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_41 snapshot 2024-03-11T23:05:53.160 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:05:53.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_41 2024-03-11T23:05:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_41': 2024-03-11T23:05:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:05:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:05:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:05:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 13fd1f990e2d 2024-03-11T23:05:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.13fd1f990e2d 2024-03-11T23:05:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:05:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:05:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:05:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:05:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:05:52 2024 2024-03-11T23:05:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:05:52 2024 2024-03-11T23:05:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:05:52 2024 2024-03-11T23:05:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:05:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:05:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 89993b1a-4069-4b34-96ea-7bed7be0e98e 2024-03-11T23:05:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:05:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:05:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_41 2024-03-11T23:05:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:05:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:05:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_41 2024-03-11T23:05:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:05:53.206 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=13 2024-03-11T23:05:53.207 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:05:53.207 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:05:53.464 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 74951680 bytes 2024-03-11T23:05:53.629 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 91175936 bytes 2024-03-11T23:05:53.772 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 117042176 bytes 2024-03-11T23:05:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 197358592 bytes 2024-03-11T23:05:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 230665216 bytes 2024-03-11T23:05:54.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 272148480 bytes 2024-03-11T23:05:54.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 329652224 bytes 2024-03-11T23:05:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 341756928 bytes 2024-03-11T23:05:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 360902656 bytes 2024-03-11T23:05:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 377161728 bytes 2024-03-11T23:05:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 416801792 bytes 2024-03-11T23:05:55.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 459285504 bytes 2024-03-11T23:05:55.467 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 467163136 bytes 2024-03-11T23:05:55.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 508032000 bytes 2024-03-11T23:05:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 552736768 bytes 2024-03-11T23:05:56.038 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 602309632 bytes 2024-03-11T23:05:56.288 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 678410240 bytes 2024-03-11T23:05:56.556 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 758280192 bytes 2024-03-11T23:05:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 852518912 bytes 2024-03-11T23:05:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 901090304 bytes 2024-03-11T23:05:57.174 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 921115648 bytes 2024-03-11T23:05:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 983448576 bytes 2024-03-11T23:05:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 1058243584 bytes 2024-03-11T23:05:57.733 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 1061083136 bytes 2024-03-11T23:05:57.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1098286080 bytes 2024-03-11T23:05:58.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1128320000 bytes 2024-03-11T23:05:58.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1161284608 bytes 2024-03-11T23:05:58.393 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1199020032 bytes 2024-03-11T23:05:58.704 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1297130496 bytes 2024-03-11T23:05:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1345372160 bytes 2024-03-11T23:05:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1425513472 bytes 2024-03-11T23:05:59.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1467147264 bytes 2024-03-11T23:05:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1492974592 bytes 2024-03-11T23:05:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1550845952 bytes 2024-03-11T23:05:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1564891136 bytes 2024-03-11T23:06:00.015 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1610856448 bytes 2024-03-11T23:06:00.211 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1634162688 bytes 2024-03-11T23:06:00.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1658094592 bytes 2024-03-11T23:06:00.537 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1701452800 bytes 2024-03-11T23:06:00.722 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1742150656 bytes 2024-03-11T23:06:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1775329280 bytes 2024-03-11T23:06:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1823076352 bytes 2024-03-11T23:06:01.298 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1892443136 bytes 2024-03-11T23:06:01.445 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 1899855872 bytes 2024-03-11T23:06:01.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 1953252352 bytes 2024-03-11T23:06:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 1973363712 bytes 2024-03-11T23:06:02.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 2025315328 bytes 2024-03-11T23:06:02.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 2079654912 bytes 2024-03-11T23:06:02.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 2090520576 bytes 2024-03-11T23:06:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 2118328320 bytes 2024-03-11T23:06:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5100 IOs, 2164848640 bytes 2024-03-11T23:06:02.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5200 IOs, 2235675648 bytes 2024-03-11T23:06:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5300 IOs, 2280366080 bytes 2024-03-11T23:06:03.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5400 IOs, 2325221376 bytes 2024-03-11T23:06:03.562 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5500 IOs, 2395047936 bytes 2024-03-11T23:06:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5600 IOs, 2450171904 bytes 2024-03-11T23:06:04.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5700 IOs, 2567824384 bytes 2024-03-11T23:06:04.256 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5800 IOs, 2588496896 bytes 2024-03-11T23:06:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5900 IOs, 2667726848 bytes 2024-03-11T23:06:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6000 IOs, 2727183360 bytes 2024-03-11T23:06:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6100 IOs, 2752861184 bytes 2024-03-11T23:06:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6200 IOs, 2779399168 bytes 2024-03-11T23:06:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6300 IOs, 2817184768 bytes 2024-03-11T23:06:05.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6400 IOs, 2877122560 bytes 2024-03-11T23:06:05.662 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6500 IOs, 2940153856 bytes 2024-03-11T23:06:05.790 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6600 IOs, 2953165824 bytes 2024-03-11T23:06:06.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6700 IOs, 3033569280 bytes 2024-03-11T23:06:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:06:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:06:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:06:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:06:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:06:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_42 2024-03-11T23:06:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 snapshot 128M 2024-03-11T23:06:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:06:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:06:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_42 2024-03-11T23:06:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:06:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:06:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_42 128M 2024-03-11T23:06:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:06:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:06:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_42 2024-03-11T23:06:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:06:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:06:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:06:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_42 2024-03-11T23:06:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:06:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:06:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:06:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:06:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_42 snapshot 2024-03-11T23:06:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:06:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:06:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_42 2024-03-11T23:06:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:06:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_42 snapshot 2024-03-11T23:06:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:06:07.286 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_42 2024-03-11T23:06:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_42': 2024-03-11T23:06:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:06:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:06:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:06:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 14099fd166a1 2024-03-11T23:06:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.14099fd166a1 2024-03-11T23:06:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:06:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:06:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:06:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:06:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:06:06 2024 2024-03-11T23:06:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:06:06 2024 2024-03-11T23:06:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:06:06 2024 2024-03-11T23:06:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:06:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:06:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: f25a7039-095a-4250-884b-b71338765a8a 2024-03-11T23:06:07.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:06:07.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:06:07.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_42 2024-03-11T23:06:07.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:06:07.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:06:07.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_42 2024-03-11T23:06:07.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:06:07.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=14 2024-03-11T23:06:07.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:06:07.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:06:07.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 21971968 bytes 2024-03-11T23:06:07.632 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 36150272 bytes 2024-03-11T23:06:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 40916992 bytes 2024-03-11T23:06:07.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 61833216 bytes 2024-03-11T23:06:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 120472576 bytes 2024-03-11T23:06:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 167991296 bytes 2024-03-11T23:06:08.459 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 202172416 bytes 2024-03-11T23:06:08.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 229989376 bytes 2024-03-11T23:06:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 268843008 bytes 2024-03-11T23:06:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 328059904 bytes 2024-03-11T23:06:09.110 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 328663040 bytes 2024-03-11T23:06:09.322 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 386498560 bytes 2024-03-11T23:06:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 396939264 bytes 2024-03-11T23:06:09.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 441079808 bytes 2024-03-11T23:06:09.848 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 492408832 bytes 2024-03-11T23:06:10.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 526363648 bytes 2024-03-11T23:06:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 586101760 bytes 2024-03-11T23:06:10.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 679349248 bytes 2024-03-11T23:06:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 709951488 bytes 2024-03-11T23:06:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 711036928 bytes 2024-03-11T23:06:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 763860992 bytes 2024-03-11T23:06:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 813465600 bytes 2024-03-11T23:06:11.412 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 854841344 bytes 2024-03-11T23:06:11.530 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 880490496 bytes 2024-03-11T23:06:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 895510528 bytes 2024-03-11T23:06:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 899610624 bytes 2024-03-11T23:06:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 977696768 bytes 2024-03-11T23:06:12.256 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1030599680 bytes 2024-03-11T23:06:12.439 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1075076096 bytes 2024-03-11T23:06:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1080997888 bytes 2024-03-11T23:06:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1131502592 bytes 2024-03-11T23:06:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1142623232 bytes 2024-03-11T23:06:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1184650240 bytes 2024-03-11T23:06:13.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1236277248 bytes 2024-03-11T23:06:13.460 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1271317504 bytes 2024-03-11T23:06:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1281060864 bytes 2024-03-11T23:06:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1285627904 bytes 2024-03-11T23:06:13.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1377835008 bytes 2024-03-11T23:06:14.147 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1423170560 bytes 2024-03-11T23:06:14.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1464241152 bytes 2024-03-11T23:06:14.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1571281920 bytes 2024-03-11T23:06:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1631818752 bytes 2024-03-11T23:06:14.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1632429056 bytes 2024-03-11T23:06:15.131 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 1664405504 bytes 2024-03-11T23:06:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 1687836672 bytes 2024-03-11T23:06:15.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 1747423232 bytes 2024-03-11T23:06:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 1815352320 bytes 2024-03-11T23:06:15.799 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.2982539677514304) > weight (0.25) 2024-03-11T23:06:15.799 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.911244030076655) > weight (0.25) 2024-03-11T23:06:15.799 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.3749877073795711) > weight (0.25) 2024-03-11T23:06:15.799 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.28068090252742706) > weight (0.25) 2024-03-11T23:06:15.800 INFO:tasks.rbd_mirror_thrash:waiting for 61 secs before thrashing 2024-03-11T23:06:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 1867149312 bytes 2024-03-11T23:06:16.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 1876108288 bytes 2024-03-11T23:06:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 1904051200 bytes 2024-03-11T23:06:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5100 IOs, 1923903488 bytes 2024-03-11T23:06:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5200 IOs, 1956261888 bytes 2024-03-11T23:06:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5300 IOs, 2004346880 bytes 2024-03-11T23:06:16.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5400 IOs, 2051704832 bytes 2024-03-11T23:06:17.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5500 IOs, 2094333952 bytes 2024-03-11T23:06:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5600 IOs, 2107850752 bytes 2024-03-11T23:06:17.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5700 IOs, 2138214400 bytes 2024-03-11T23:06:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5800 IOs, 2161117184 bytes 2024-03-11T23:06:17.704 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7869018228567501) > weight (0.25) 2024-03-11T23:06:17.705 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.4489754469374685) > weight (0.25) 2024-03-11T23:06:17.705 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-03-11T23:06:17.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-03-11T23:06:17.705 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-03-11T23:06:17.705 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-03-11T23:06:17.705 INFO:teuthology.orchestra.run:waiting for 600 2024-03-11T23:06:17.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi132.stderr:2024-03-11T23:06:17.702+0000 7fd2ba3ab640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 52104) UID: 1000 2024-03-11T23:06:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5900 IOs, 2212692992 bytes 2024-03-11T23:06:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6000 IOs, 2274281472 bytes 2024-03-11T23:06:18.173 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6100 IOs, 2334398464 bytes 2024-03-11T23:06:18.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6200 IOs, 2369431552 bytes 2024-03-11T23:06:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6300 IOs, 2371979264 bytes 2024-03-11T23:06:18.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6400 IOs, 2424897536 bytes 2024-03-11T23:06:18.818 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6500 IOs, 2463240192 bytes 2024-03-11T23:06:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6600 IOs, 2473060352 bytes 2024-03-11T23:06:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6700 IOs, 2496547840 bytes 2024-03-11T23:06:19.281 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6800 IOs, 2543218688 bytes 2024-03-11T23:06:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6900 IOs, 2597156864 bytes 2024-03-11T23:06:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7000 IOs, 2597774336 bytes 2024-03-11T23:06:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7100 IOs, 2651538432 bytes 2024-03-11T23:06:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7200 IOs, 2707165184 bytes 2024-03-11T23:06:20.122 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7300 IOs, 2727235584 bytes 2024-03-11T23:06:20.310 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7400 IOs, 2753223680 bytes 2024-03-11T23:06:20.477 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7500 IOs, 2798036992 bytes 2024-03-11T23:06:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7600 IOs, 2799421440 bytes 2024-03-11T23:06:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7700 IOs, 2821357568 bytes 2024-03-11T23:06:20.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7800 IOs, 2862448640 bytes 2024-03-11T23:06:21.130 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7900 IOs, 2899309568 bytes 2024-03-11T23:06:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:8000 IOs, 2934793216 bytes 2024-03-11T23:06:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:06:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:06:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:06:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:06:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:06:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_43 2024-03-11T23:06:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 snapshot 128M 2024-03-11T23:06:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:06:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:06:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_43 2024-03-11T23:06:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:06:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:06:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_43 128M 2024-03-11T23:06:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:06:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:06:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_43 2024-03-11T23:06:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:06:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:06:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:06:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_43 2024-03-11T23:06:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:06:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:06:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:06:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:06:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_43 snapshot 2024-03-11T23:06:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:06:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:06:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_43 2024-03-11T23:06:21.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:06:21.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_43 snapshot 2024-03-11T23:06:21.899 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:06:21.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_43 2024-03-11T23:06:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_43': 2024-03-11T23:06:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:06:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:06:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:06:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 14152443122c 2024-03-11T23:06:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.14152443122c 2024-03-11T23:06:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:06:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:06:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:06:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:06:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:06:21 2024 2024-03-11T23:06:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:06:21 2024 2024-03-11T23:06:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:06:21 2024 2024-03-11T23:06:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:06:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:06:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: a46955d2-b05b-472d-bb17-68f3cabe31e2 2024-03-11T23:06:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:06:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:06:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_43 2024-03-11T23:06:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:06:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:06:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_43 2024-03-11T23:06:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:06:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=6 2024-03-11T23:06:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:06:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:06:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 102169600 bytes 2024-03-11T23:06:22.455 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 139843584 bytes 2024-03-11T23:06:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 167266304 bytes 2024-03-11T23:06:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 198256640 bytes 2024-03-11T23:06:22.908 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 204930048 bytes 2024-03-11T23:06:23.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 211986432 bytes 2024-03-11T23:06:23.200 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 251680768 bytes 2024-03-11T23:06:23.394 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 297052160 bytes 2024-03-11T23:06:23.514 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 310669312 bytes 2024-03-11T23:06:23.615 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 326104064 bytes 2024-03-11T23:06:23.807 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-03-11T23:06:23.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-03-11T23:06:23.807 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-03-11T23:06:23.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-03-11T23:06:23.810 INFO:tasks.rbd_mirror_thrash:waiting for 64 secs before thrashing 2024-03-11T23:06:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 387490816 bytes 2024-03-11T23:06:24.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 470090752 bytes 2024-03-11T23:06:24.310 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 494823424 bytes 2024-03-11T23:06:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 558794752 bytes 2024-03-11T23:06:24.727 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 591237120 bytes 2024-03-11T23:06:24.864 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 609544192 bytes 2024-03-11T23:06:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 658419712 bytes 2024-03-11T23:06:25.210 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 682392576 bytes 2024-03-11T23:06:25.359 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 710674432 bytes 2024-03-11T23:06:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 779157504 bytes 2024-03-11T23:06:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 812492800 bytes 2024-03-11T23:06:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 845981696 bytes 2024-03-11T23:06:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 926912512 bytes 2024-03-11T23:06:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 961961984 bytes 2024-03-11T23:06:26.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1013510144 bytes 2024-03-11T23:06:26.613 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1024613376 bytes 2024-03-11T23:06:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1060820992 bytes 2024-03-11T23:06:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1107339264 bytes 2024-03-11T23:06:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1120892928 bytes 2024-03-11T23:06:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1173979136 bytes 2024-03-11T23:06:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1227046912 bytes 2024-03-11T23:06:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1307410432 bytes 2024-03-11T23:06:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:06:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:06:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:06:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:06:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:06:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_44 2024-03-11T23:06:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 snapshot 128M 2024-03-11T23:06:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:06:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:06:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_44 2024-03-11T23:06:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:06:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:06:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_44 128M 2024-03-11T23:06:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:06:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:06:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_44 2024-03-11T23:06:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:06:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:06:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:06:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_44 2024-03-11T23:06:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:06:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:06:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:06:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:06:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_44 snapshot 2024-03-11T23:06:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:06:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:06:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_44 2024-03-11T23:06:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:06:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_44 snapshot 2024-03-11T23:06:28.931 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:06:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_44 2024-03-11T23:06:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_44': 2024-03-11T23:06:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:06:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:06:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:06:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 1427a24e9f79 2024-03-11T23:06:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.1427a24e9f79 2024-03-11T23:06:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:06:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:06:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:06:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:06:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:06:27 2024 2024-03-11T23:06:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:06:27 2024 2024-03-11T23:06:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:06:27 2024 2024-03-11T23:06:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:06:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:06:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 5772827a-fff4-4729-bef7-03d63872ad43 2024-03-11T23:06:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:06:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:06:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_44 2024-03-11T23:06:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:06:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:06:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_44 2024-03-11T23:06:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:06:28.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=10 2024-03-11T23:06:28.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:06:28.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:06:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 30792704 bytes 2024-03-11T23:06:29.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 120563712 bytes 2024-03-11T23:06:29.683 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 209701888 bytes 2024-03-11T23:06:29.988 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 274176000 bytes 2024-03-11T23:06:30.208 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 334001152 bytes 2024-03-11T23:06:30.412 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 388263936 bytes 2024-03-11T23:06:30.524 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 397467648 bytes 2024-03-11T23:06:30.650 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 411588608 bytes 2024-03-11T23:06:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 491144192 bytes 2024-03-11T23:06:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 510793728 bytes 2024-03-11T23:06:31.196 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 511428608 bytes 2024-03-11T23:06:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 569158656 bytes 2024-03-11T23:06:31.560 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 592155648 bytes 2024-03-11T23:06:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 706309120 bytes 2024-03-11T23:06:32.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 739613696 bytes 2024-03-11T23:06:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 762864640 bytes 2024-03-11T23:06:32.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 767536128 bytes 2024-03-11T23:06:32.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 827913216 bytes 2024-03-11T23:06:32.647 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 845336576 bytes 2024-03-11T23:06:32.896 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 909970432 bytes 2024-03-11T23:06:32.999 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 914721792 bytes 2024-03-11T23:06:33.195 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 956704768 bytes 2024-03-11T23:06:33.331 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 973639680 bytes 2024-03-11T23:06:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 1022297088 bytes 2024-03-11T23:06:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1083795456 bytes 2024-03-11T23:06:33.999 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1156641792 bytes 2024-03-11T23:06:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1176458240 bytes 2024-03-11T23:06:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1209364480 bytes 2024-03-11T23:06:34.492 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1250362368 bytes 2024-03-11T23:06:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1281561600 bytes 2024-03-11T23:06:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1355404288 bytes 2024-03-11T23:06:35.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1428658176 bytes 2024-03-11T23:06:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1450157056 bytes 2024-03-11T23:06:35.455 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1486432256 bytes 2024-03-11T23:06:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1522717696 bytes 2024-03-11T23:06:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1523356672 bytes 2024-03-11T23:06:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1560698880 bytes 2024-03-11T23:06:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1649359872 bytes 2024-03-11T23:06:36.286 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1673383936 bytes 2024-03-11T23:06:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1698413568 bytes 2024-03-11T23:06:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1762103296 bytes 2024-03-11T23:06:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1811155968 bytes 2024-03-11T23:06:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1877338112 bytes 2024-03-11T23:06:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 1909122048 bytes 2024-03-11T23:06:37.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 1929897984 bytes 2024-03-11T23:06:37.711 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 2024148992 bytes 2024-03-11T23:06:38.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 2098184192 bytes 2024-03-11T23:06:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 2109989888 bytes 2024-03-11T23:06:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 2159820800 bytes 2024-03-11T23:06:38.518 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 2209017856 bytes 2024-03-11T23:06:38.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5100 IOs, 2253906944 bytes 2024-03-11T23:06:38.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5200 IOs, 2264298496 bytes 2024-03-11T23:06:38.988 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:06:38.988 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:06:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:06:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:06:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:06:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_45 2024-03-11T23:06:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 snapshot 128M 2024-03-11T23:06:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:06:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:06:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_45 2024-03-11T23:06:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:06:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:06:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_45 128M 2024-03-11T23:06:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:06:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:06:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_45 2024-03-11T23:06:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:06:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:06:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:06:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_45 2024-03-11T23:06:39.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:06:39.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:06:39.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:06:39.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:06:39.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_45 snapshot 2024-03-11T23:06:39.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:06:39.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:06:39.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_45 2024-03-11T23:06:39.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:06:39.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_45 snapshot 2024-03-11T23:06:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:06:40.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_45 2024-03-11T23:06:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_45': 2024-03-11T23:06:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:06:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:06:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:06:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 1433b4f452f3 2024-03-11T23:06:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.1433b4f452f3 2024-03-11T23:06:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:06:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:06:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:06:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:06:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:06:39 2024 2024-03-11T23:06:40.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:06:39 2024 2024-03-11T23:06:40.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:06:39 2024 2024-03-11T23:06:40.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:06:40.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:06:40.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: d3a33f3e-a3e4-4e65-b3f9-eb5141260b1a 2024-03-11T23:06:40.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:06:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:06:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_45 2024-03-11T23:06:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:06:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:06:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_45 2024-03-11T23:06:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:06:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=6 2024-03-11T23:06:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:06:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:06:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 48559104 bytes 2024-03-11T23:06:40.509 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 104997888 bytes 2024-03-11T23:06:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 105612288 bytes 2024-03-11T23:06:40.702 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 106414080 bytes 2024-03-11T23:06:40.829 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 118737920 bytes 2024-03-11T23:06:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 175878144 bytes 2024-03-11T23:06:41.217 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 211128320 bytes 2024-03-11T23:06:41.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 222021632 bytes 2024-03-11T23:06:41.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 273527808 bytes 2024-03-11T23:06:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 327815168 bytes 2024-03-11T23:06:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 364909568 bytes 2024-03-11T23:06:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 374887424 bytes 2024-03-11T23:06:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 456227840 bytes 2024-03-11T23:06:42.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 456878080 bytes 2024-03-11T23:06:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 497361920 bytes 2024-03-11T23:06:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 529176576 bytes 2024-03-11T23:06:42.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 577329152 bytes 2024-03-11T23:06:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 661176320 bytes 2024-03-11T23:06:43.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 661807104 bytes 2024-03-11T23:06:43.537 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 697036800 bytes 2024-03-11T23:06:43.766 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 758178816 bytes 2024-03-11T23:06:43.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 813699072 bytes 2024-03-11T23:06:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 897359872 bytes 2024-03-11T23:06:44.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 911805440 bytes 2024-03-11T23:06:44.488 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 926171136 bytes 2024-03-11T23:06:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 990843904 bytes 2024-03-11T23:06:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1006799872 bytes 2024-03-11T23:06:44.985 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1035239424 bytes 2024-03-11T23:06:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1068898304 bytes 2024-03-11T23:06:45.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1120510976 bytes 2024-03-11T23:06:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1174136832 bytes 2024-03-11T23:06:45.792 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1228483584 bytes 2024-03-11T23:06:45.985 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1275214848 bytes 2024-03-11T23:06:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:06:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:06:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:06:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:06:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:06:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_46 2024-03-11T23:06:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 snapshot 128M 2024-03-11T23:06:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:06:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:06:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_46 2024-03-11T23:06:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:06:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:06:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_46 128M 2024-03-11T23:06:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:06:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:06:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_46 2024-03-11T23:06:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:06:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:06:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:06:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_46 2024-03-11T23:06:46.108 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:06:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:06:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:06:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:06:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_46 snapshot 2024-03-11T23:06:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:06:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:06:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_46 2024-03-11T23:06:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:06:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_46 snapshot 2024-03-11T23:06:47.239 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:06:47.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_46 2024-03-11T23:06:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_46': 2024-03-11T23:06:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:06:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:06:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:06:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 143fd59aacd0 2024-03-11T23:06:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.143fd59aacd0 2024-03-11T23:06:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:06:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:06:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:06:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:06:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:06:46 2024 2024-03-11T23:06:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:06:46 2024 2024-03-11T23:06:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:06:46 2024 2024-03-11T23:06:47.286 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:06:47.286 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:06:47.286 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: b3239d40-57ce-462a-97ff-5a0055d40fa0 2024-03-11T23:06:47.286 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:06:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:06:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_46 2024-03-11T23:06:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:06:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:06:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_46 2024-03-11T23:06:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:06:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=5 2024-03-11T23:06:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:06:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:06:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 20860928 bytes 2024-03-11T23:06:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 66985984 bytes 2024-03-11T23:06:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 115469312 bytes 2024-03-11T23:06:47.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 116075520 bytes 2024-03-11T23:06:48.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 193648640 bytes 2024-03-11T23:06:48.414 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 239024128 bytes 2024-03-11T23:06:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 296754176 bytes 2024-03-11T23:06:48.841 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 349136896 bytes 2024-03-11T23:06:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 368246784 bytes 2024-03-11T23:06:49.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 368875520 bytes 2024-03-11T23:06:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 377255936 bytes 2024-03-11T23:06:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 455134208 bytes 2024-03-11T23:06:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 489427968 bytes 2024-03-11T23:06:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 556100608 bytes 2024-03-11T23:06:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 612063232 bytes 2024-03-11T23:06:50.081 INFO:tasks.ceph.cluster2.osd.2.smithi132.stderr:2024-03-11T23:06:50.077+0000 7f6376022640 -1 reset not still connected to 0x5650825168f0 2024-03-11T23:06:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 640398336 bytes 2024-03-11T23:06:50.400 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 683206656 bytes 2024-03-11T23:06:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 717658112 bytes 2024-03-11T23:06:50.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 761049088 bytes 2024-03-11T23:06:50.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 809252864 bytes 2024-03-11T23:06:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 868656128 bytes 2024-03-11T23:06:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 872812544 bytes 2024-03-11T23:06:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 900639744 bytes 2024-03-11T23:06:51.652 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 955326464 bytes 2024-03-11T23:06:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 981351424 bytes 2024-03-11T23:06:51.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 998957056 bytes 2024-03-11T23:06:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1038925824 bytes 2024-03-11T23:06:52.289 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1087285248 bytes 2024-03-11T23:06:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:06:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:06:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:06:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:06:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:06:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_47 2024-03-11T23:06:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 snapshot 128M 2024-03-11T23:06:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:06:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:06:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_47 2024-03-11T23:06:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:06:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:06:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_47 128M 2024-03-11T23:06:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:06:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:06:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_47 2024-03-11T23:06:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:06:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:06:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:06:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:06:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_47 2024-03-11T23:06:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:06:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:06:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:06:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:06:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_47 snapshot 2024-03-11T23:06:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:06:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:06:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_47 2024-03-11T23:06:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:06:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_47 snapshot 2024-03-11T23:06:53.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:06:53.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_47 2024-03-11T23:06:53.119 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_47': 2024-03-11T23:06:53.119 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:06:53.119 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:06:53.119 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:06:53.119 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 144bb117b762 2024-03-11T23:06:53.119 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.144bb117b762 2024-03-11T23:06:53.119 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:06:53.119 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:06:53.119 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:06:53.119 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:06:53.120 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:06:52 2024 2024-03-11T23:06:53.120 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:06:52 2024 2024-03-11T23:06:53.120 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:06:52 2024 2024-03-11T23:06:53.120 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:06:53.120 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:06:53.120 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 692e2de5-49a3-485f-a3e3-7e5e2a4ce76d 2024-03-11T23:06:53.120 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:06:53.122 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:06:53.122 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_47 2024-03-11T23:06:53.122 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:06:53.123 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:06:53.123 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_47 2024-03-11T23:06:53.123 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:06:53.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=11 2024-03-11T23:06:53.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:06:53.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:06:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 68637696 bytes 2024-03-11T23:06:53.492 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 73272320 bytes 2024-03-11T23:06:53.670 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 115447808 bytes 2024-03-11T23:06:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 159774720 bytes 2024-03-11T23:06:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 199122944 bytes 2024-03-11T23:06:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 270676992 bytes 2024-03-11T23:06:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 286518272 bytes 2024-03-11T23:06:54.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 310522880 bytes 2024-03-11T23:06:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 363672576 bytes 2024-03-11T23:06:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 381630464 bytes 2024-03-11T23:06:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 427606016 bytes 2024-03-11T23:06:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 462292992 bytes 2024-03-11T23:06:55.460 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 509386752 bytes 2024-03-11T23:06:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 509989888 bytes 2024-03-11T23:06:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 558782464 bytes 2024-03-11T23:06:55.908 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 565894144 bytes 2024-03-11T23:06:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 607484928 bytes 2024-03-11T23:06:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 611721216 bytes 2024-03-11T23:06:56.446 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 684790784 bytes 2024-03-11T23:06:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 713789440 bytes 2024-03-11T23:06:56.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 752832512 bytes 2024-03-11T23:06:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 753443840 bytes 2024-03-11T23:06:57.017 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 754044928 bytes 2024-03-11T23:06:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 825450496 bytes 2024-03-11T23:06:57.403 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 848058368 bytes 2024-03-11T23:06:57.559 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 880057344 bytes 2024-03-11T23:06:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 973238272 bytes 2024-03-11T23:06:58.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1041179648 bytes 2024-03-11T23:06:58.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1135703040 bytes 2024-03-11T23:06:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1216405504 bytes 2024-03-11T23:06:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1314515968 bytes 2024-03-11T23:06:59.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1365143552 bytes 2024-03-11T23:06:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1385887744 bytes 2024-03-11T23:06:59.399 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1392962560 bytes 2024-03-11T23:06:59.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1437172736 bytes 2024-03-11T23:06:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1476288512 bytes 2024-03-11T23:06:59.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1536470016 bytes 2024-03-11T23:07:00.167 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1589522432 bytes 2024-03-11T23:07:00.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1631152128 bytes 2024-03-11T23:07:00.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1674017792 bytes 2024-03-11T23:07:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1760842752 bytes 2024-03-11T23:07:01.065 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1809230848 bytes 2024-03-11T23:07:01.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1838397440 bytes 2024-03-11T23:07:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 1864883200 bytes 2024-03-11T23:07:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 1868726272 bytes 2024-03-11T23:07:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 1906054144 bytes 2024-03-11T23:07:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 2030435328 bytes 2024-03-11T23:07:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 2084215808 bytes 2024-03-11T23:07:02.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 2091778048 bytes 2024-03-11T23:07:02.604 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 2180186112 bytes 2024-03-11T23:07:02.802 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5100 IOs, 2226465792 bytes 2024-03-11T23:07:03.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5200 IOs, 2289484800 bytes 2024-03-11T23:07:03.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5300 IOs, 2329350144 bytes 2024-03-11T23:07:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5400 IOs, 2353936384 bytes 2024-03-11T23:07:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5500 IOs, 2370411520 bytes 2024-03-11T23:07:03.721 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5600 IOs, 2432635904 bytes 2024-03-11T23:07:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5700 IOs, 2537073664 bytes 2024-03-11T23:07:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:07:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:07:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:07:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:07:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:07:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_48 2024-03-11T23:07:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 snapshot 128M 2024-03-11T23:07:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:07:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:07:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:07:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:07:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_48 2024-03-11T23:07:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:07:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:07:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:07:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:07:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_48 128M 2024-03-11T23:07:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:07:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:07:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:07:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:07:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_48 2024-03-11T23:07:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:07:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:07:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:07:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:07:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:07:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_48 2024-03-11T23:07:04.180 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:07:04.180 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:07:04.180 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:07:04.180 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:07:04.180 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_48 snapshot 2024-03-11T23:07:04.180 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:07:04.181 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:07:04.181 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_48 2024-03-11T23:07:04.181 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:07:04.181 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_48 snapshot 2024-03-11T23:07:05.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:07:05.196 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_48 2024-03-11T23:07:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_48': 2024-03-11T23:07:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:07:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:07:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:07:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 1457a0525265 2024-03-11T23:07:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.1457a0525265 2024-03-11T23:07:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:07:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:07:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:07:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:07:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:07:04 2024 2024-03-11T23:07:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:07:04 2024 2024-03-11T23:07:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:07:04 2024 2024-03-11T23:07:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:07:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:07:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: d3f7cb01-e07f-48c5-9eba-cb65593159c2 2024-03-11T23:07:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:07:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:07:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_48 2024-03-11T23:07:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:07:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:07:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_48 2024-03-11T23:07:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:07:05.238 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=13 2024-03-11T23:07:05.238 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:07:05.238 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:07:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 57489408 bytes 2024-03-11T23:07:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 97377280 bytes 2024-03-11T23:07:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 147505152 bytes 2024-03-11T23:07:06.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 185793536 bytes 2024-03-11T23:07:06.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 210176000 bytes 2024-03-11T23:07:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 271680512 bytes 2024-03-11T23:07:06.520 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 273038336 bytes 2024-03-11T23:07:06.669 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 297488384 bytes 2024-03-11T23:07:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 316803072 bytes 2024-03-11T23:07:07.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 358049792 bytes 2024-03-11T23:07:07.157 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 358672384 bytes 2024-03-11T23:07:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 411143168 bytes 2024-03-11T23:07:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 455576576 bytes 2024-03-11T23:07:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 500981760 bytes 2024-03-11T23:07:07.925 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 522380288 bytes 2024-03-11T23:07:08.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 618219520 bytes 2024-03-11T23:07:08.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 641971200 bytes 2024-03-11T23:07:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 642574336 bytes 2024-03-11T23:07:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 721537024 bytes 2024-03-11T23:07:08.920 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 766707712 bytes 2024-03-11T23:07:09.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 816172032 bytes 2024-03-11T23:07:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 846601216 bytes 2024-03-11T23:07:09.417 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 865867776 bytes 2024-03-11T23:07:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 914296832 bytes 2024-03-11T23:07:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1023694848 bytes 2024-03-11T23:07:10.103 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1052063744 bytes 2024-03-11T23:07:10.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1099430912 bytes 2024-03-11T23:07:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1161459712 bytes 2024-03-11T23:07:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1191701504 bytes 2024-03-11T23:07:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1275314176 bytes 2024-03-11T23:07:11.129 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1321991168 bytes 2024-03-11T23:07:11.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1363468288 bytes 2024-03-11T23:07:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1428703232 bytes 2024-03-11T23:07:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1482600448 bytes 2024-03-11T23:07:11.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1506622464 bytes 2024-03-11T23:07:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1537344512 bytes 2024-03-11T23:07:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1576059904 bytes 2024-03-11T23:07:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1660863488 bytes 2024-03-11T23:07:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1671396352 bytes 2024-03-11T23:07:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1730187264 bytes 2024-03-11T23:07:13.029 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1765696512 bytes 2024-03-11T23:07:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1816949760 bytes 2024-03-11T23:07:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1829506048 bytes 2024-03-11T23:07:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 1869566976 bytes 2024-03-11T23:07:13.910 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 1901243392 bytes 2024-03-11T23:07:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 1948267520 bytes 2024-03-11T23:07:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 1970138112 bytes 2024-03-11T23:07:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 1974342656 bytes 2024-03-11T23:07:14.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 2035255296 bytes 2024-03-11T23:07:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 2054206464 bytes 2024-03-11T23:07:14.911 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5100 IOs, 2097067008 bytes 2024-03-11T23:07:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5200 IOs, 2119830528 bytes 2024-03-11T23:07:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5300 IOs, 2172484608 bytes 2024-03-11T23:07:15.576 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5400 IOs, 2282955776 bytes 2024-03-11T23:07:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5500 IOs, 2323410944 bytes 2024-03-11T23:07:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5600 IOs, 2367184896 bytes 2024-03-11T23:07:16.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5700 IOs, 2404410368 bytes 2024-03-11T23:07:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5800 IOs, 2424547328 bytes 2024-03-11T23:07:16.525 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5900 IOs, 2471447552 bytes 2024-03-11T23:07:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6000 IOs, 2513510400 bytes 2024-03-11T23:07:16.800 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.31786653414451993) > weight (0.25) 2024-03-11T23:07:16.800 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-03-11T23:07:16.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-03-11T23:07:16.801 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-03-11T23:07:16.801 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-03-11T23:07:16.801 INFO:teuthology.orchestra.run:waiting for 600 2024-03-11T23:07:16.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi132.stderr:2024-03-11T23:07:16.797+0000 7f3f50093640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 46147) UID: 1000 2024-03-11T23:07:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6100 IOs, 2538620928 bytes 2024-03-11T23:07:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6200 IOs, 2566263808 bytes 2024-03-11T23:07:17.174 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6300 IOs, 2577784832 bytes 2024-03-11T23:07:17.302 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6400 IOs, 2590509056 bytes 2024-03-11T23:07:17.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6500 IOs, 2624969728 bytes 2024-03-11T23:07:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6600 IOs, 2730774528 bytes 2024-03-11T23:07:17.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6700 IOs, 2767874048 bytes 2024-03-11T23:07:18.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6800 IOs, 2794910720 bytes 2024-03-11T23:07:18.244 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:07:18.244 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:07:18.244 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:07:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:07:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:07:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_49 2024-03-11T23:07:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 snapshot 128M 2024-03-11T23:07:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:07:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:07:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:07:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:07:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_49 2024-03-11T23:07:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:07:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:07:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:07:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:07:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_49 128M 2024-03-11T23:07:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:07:18.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:07:18.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:07:18.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:07:18.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_49 2024-03-11T23:07:18.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:07:18.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:07:18.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:07:18.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:07:18.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:07:18.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_49 2024-03-11T23:07:18.291 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:07:18.291 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:07:18.291 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:07:18.291 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:07:18.291 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_49 snapshot 2024-03-11T23:07:18.291 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:07:18.291 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:07:18.291 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_49 2024-03-11T23:07:18.291 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:07:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_49 snapshot 2024-03-11T23:07:19.286 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:07:19.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_49 2024-03-11T23:07:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_49': 2024-03-11T23:07:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:07:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:07:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:07:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 1462ffd876cd 2024-03-11T23:07:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.1462ffd876cd 2024-03-11T23:07:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:07:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:07:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:07:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:07:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:07:18 2024 2024-03-11T23:07:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:07:18 2024 2024-03-11T23:07:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:07:18 2024 2024-03-11T23:07:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:07:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:07:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 71ab5b7e-440a-48ab-8f45-bb970a72ca7e 2024-03-11T23:07:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:07:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:07:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_49 2024-03-11T23:07:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:07:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:07:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_49 2024-03-11T23:07:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:07:19.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=6 2024-03-11T23:07:19.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:07:19.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:07:19.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 34666496 bytes 2024-03-11T23:07:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 48783360 bytes 2024-03-11T23:07:19.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 74264576 bytes 2024-03-11T23:07:19.975 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 94873600 bytes 2024-03-11T23:07:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 96102400 bytes 2024-03-11T23:07:20.273 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 138858496 bytes 2024-03-11T23:07:20.401 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 153910272 bytes 2024-03-11T23:07:20.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 189089792 bytes 2024-03-11T23:07:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 246231040 bytes 2024-03-11T23:07:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 246839296 bytes 2024-03-11T23:07:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 288766976 bytes 2024-03-11T23:07:21.282 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 336630784 bytes 2024-03-11T23:07:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 353784832 bytes 2024-03-11T23:07:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 377820160 bytes 2024-03-11T23:07:21.723 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 428173312 bytes 2024-03-11T23:07:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 457389056 bytes 2024-03-11T23:07:22.170 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 535804928 bytes 2024-03-11T23:07:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 685057024 bytes 2024-03-11T23:07:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 705876992 bytes 2024-03-11T23:07:22.902 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-03-11T23:07:22.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-03-11T23:07:22.903 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-03-11T23:07:22.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-03-11T23:07:22.905 INFO:tasks.rbd_mirror_thrash:waiting for 63 secs before thrashing 2024-03-11T23:07:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 773989376 bytes 2024-03-11T23:07:23.108 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 789627904 bytes 2024-03-11T23:07:23.296 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 826969088 bytes 2024-03-11T23:07:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 926353408 bytes 2024-03-11T23:07:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 968205312 bytes 2024-03-11T23:07:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1024465920 bytes 2024-03-11T23:07:24.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1061349376 bytes 2024-03-11T23:07:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1061963776 bytes 2024-03-11T23:07:24.530 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1129356288 bytes 2024-03-11T23:07:24.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1156108288 bytes 2024-03-11T23:07:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1188965376 bytes 2024-03-11T23:07:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1250672640 bytes 2024-03-11T23:07:25.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1306625024 bytes 2024-03-11T23:07:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:07:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:07:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:07:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:07:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:07:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_50 2024-03-11T23:07:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 snapshot 128M 2024-03-11T23:07:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:07:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:07:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:07:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:07:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_50 2024-03-11T23:07:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:07:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:07:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n snapshot ']' 2024-03-11T23:07:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:07:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_50 128M 2024-03-11T23:07:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:07:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:07:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:07:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:07:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_50 2024-03-11T23:07:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:07:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:07:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 128M ']' 2024-03-11T23:07:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=128M 2024-03-11T23:07:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:07:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_50 2024-03-11T23:07:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ is_pool_mirror_mode_image mirror 2024-03-11T23:07:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:07:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' image = image ']' 2024-03-11T23:07:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:07:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_50 snapshot 2024-03-11T23:07:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:07:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:07:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_50 2024-03-11T23:07:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:07:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_50 snapshot 2024-03-11T23:07:26.323 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:07:26.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_50 2024-03-11T23:07:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_50': 2024-03-11T23:07:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:07:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:07:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:07:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 147519347607 2024-03-11T23:07:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.147519347607 2024-03-11T23:07:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:07:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:07:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:07:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:07:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:07:25 2024 2024-03-11T23:07:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:07:25 2024 2024-03-11T23:07:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:07:25 2024 2024-03-11T23:07:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:07:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:07:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 3fb7dde7-de15-40cd-9bc8-e3db123cf169 2024-03-11T23:07:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:07:26.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n '' ']' 2024-03-11T23:07:26.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_50 2024-03-11T23:07:26.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:07:26.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:07:26.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_50 2024-03-11T23:07:26.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:07:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=5 2024-03-11T23:07:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:07:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:07:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 33733632 bytes 2024-03-11T23:07:26.761 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 75338752 bytes 2024-03-11T23:07:26.868 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 79733760 bytes 2024-03-11T23:07:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 140300288 bytes 2024-03-11T23:07:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 152498176 bytes 2024-03-11T23:07:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 246226944 bytes 2024-03-11T23:07:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 261408768 bytes 2024-03-11T23:07:27.811 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-03-11T23:07:27.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-03-11T23:07:27.812 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-03-11T23:07:27.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi132.stderr:2024-03-11T23:07:27.810+0000 7f9f9c5ff640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 50993) UID: 1000 2024-03-11T23:07:27.869 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 345074688 bytes 2024-03-11T23:07:28.085 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 396843008 bytes 2024-03-11T23:07:28.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 469515264 bytes 2024-03-11T23:07:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 523868160 bytes 2024-03-11T23:07:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 556781568 bytes 2024-03-11T23:07:28.790 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 563631104 bytes 2024-03-11T23:07:28.928 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 580418560 bytes 2024-03-11T23:07:29.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 595565568 bytes 2024-03-11T23:07:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 630150144 bytes 2024-03-11T23:07:29.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 699506688 bytes 2024-03-11T23:07:29.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 700124160 bytes 2024-03-11T23:07:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 753229824 bytes 2024-03-11T23:07:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 773182464 bytes 2024-03-11T23:07:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 863399936 bytes 2024-03-11T23:07:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 863992832 bytes 2024-03-11T23:07:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 870115328 bytes 2024-03-11T23:07:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 919012352 bytes 2024-03-11T23:07:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 982990848 bytes 2024-03-11T23:07:31.108 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1075345408 bytes 2024-03-11T23:07:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1081210880 bytes 2024-03-11T23:07:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:07:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:07:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:07:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:07:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_pool_images cluster2 mirror 50 2024-03-11T23:07:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:07:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:07:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_count=50 2024-03-11T23:07:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:07:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local count 2024-03-11T23:07:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local last_count=0 2024-03-11T23:07:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:07:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ seq 1 40 2024-03-11T23:07:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in `seq 1 40` 2024-03-11T23:07:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 1 -ne 1 2024-03-11T23:07:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-03-11T23:07:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep 'images: ' 2024-03-11T23:07:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cut '-d ' -f 2 2024-03-11T23:07:31.414 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: failed to query services: (13) Permission denied 2024-03-11T23:07:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ count=50 2024-03-11T23:07:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 50 = 50 2024-03-11T23:07:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:07:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_pool_healthy cluster2 mirror 2024-03-11T23:07:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:07:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:07:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:07:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state 2024-03-11T23:07:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ seq 1 40 2024-03-11T23:07:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in `seq 1 40` 2024-03-11T23:07:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 1 -ne 1 2024-03-11T23:07:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-03-11T23:07:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep 'image health:' 2024-03-11T23:07:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cut '-d ' -f 3 2024-03-11T23:07:31.453 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: failed to query services: (13) Permission denied 2024-03-11T23:07:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ state=WARNING 2024-03-11T23:07:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test WARNING = ERROR 2024-03-11T23:07:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test WARNING = OK 2024-03-11T23:07:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in `seq 1 40` 2024-03-11T23:07:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 2 -ne 1 2024-03-11T23:07:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 30 2024-03-11T23:07:33.813 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-03-11T23:07:33.813 INFO:teuthology.orchestra.run:waiting for 600 2024-03-11T23:07:33.813 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-03-11T23:07:33.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-03-11T23:07:33.814 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-03-11T23:07:33.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-03-11T23:07:33.816 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2024-03-11T23:08:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-03-11T23:08:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep 'image health:' 2024-03-11T23:08:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cut '-d ' -f 3 2024-03-11T23:08:01.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: failed to query services: (13) Permission denied 2024-03-11T23:08:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ state=OK 2024-03-11T23:08:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test OK = ERROR 2024-03-11T23:08:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test OK = OK 2024-03-11T23:08:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:08:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_pool_images cluster1 mirror 50 2024-03-11T23:08:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_count=50 2024-03-11T23:08:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:08:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local count 2024-03-11T23:08:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local last_count=0 2024-03-11T23:08:01.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:08:01.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ seq 1 40 2024-03-11T23:08:01.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in `seq 1 40` 2024-03-11T23:08:01.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 1 -ne 1 2024-03-11T23:08:01.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-03-11T23:08:01.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep 'images: ' 2024-03-11T23:08:01.505 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cut '-d ' -f 2 2024-03-11T23:08:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: failed to query services: (13) Permission denied 2024-03-11T23:08:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ count=50 2024-03-11T23:08:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 50 = 50 2024-03-11T23:08:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:08:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_pool_healthy cluster1 mirror 2024-03-11T23:08:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:08:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state 2024-03-11T23:08:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ seq 1 40 2024-03-11T23:08:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in `seq 1 40` 2024-03-11T23:08:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 1 -ne 1 2024-03-11T23:08:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-03-11T23:08:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep 'image health:' 2024-03-11T23:08:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cut '-d ' -f 3 2024-03-11T23:08:01.583 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: failed to query services: (13) Permission denied 2024-03-11T23:08:01.586 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ state=OK 2024-03-11T23:08:01.587 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test OK = ERROR 2024-03-11T23:08:01.587 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test OK = OK 2024-03-11T23:08:01.587 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:08:01.587 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ testlog 'TEST: compare many images' 2024-03-11T23:08:01.587 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ tee -a /tmp/tmp.7mIVT0l9FS/rbd-mirror.test.log 2024-03-11T23:08:01.587 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ date '+%F %T' 2024-03-11T23:08:01.587 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo 2024-03-11 23:08:01 TEST: compare many images 2024-03-11T23:08:01.588 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:2024-03-11 23:08:01 TEST: compare many images 2024-03-11T23:08:01.588 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ seq 1 50 2024-03-11T23:08:01.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:08:01.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_1 2024-03-11T23:08:01.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_1 snap 2024-03-11T23:08:01.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:01.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:01.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T23:08:01.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:08:01.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:08:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2024-03-11T23:08:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T23:08:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2024-03-11T23:08:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T23:08:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:08:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:08:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:08:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:08:03.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2024-03-11T23:08:03.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:03.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:03.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T23:08:03.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:08:03.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:08:03.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:08:03.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2024-03-11T23:08:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:08:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:08:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:08:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:08:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:08:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:08:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:08:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:08:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:08:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:08:03.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:08:03.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2024-03-11T23:08:03.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:08:03.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:08:03.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:08:03.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:08:03.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:08:03.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:08:03.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:08:03.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:08:03.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:08:03.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:08:03.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:08:03.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:03.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:08:03.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:08:03.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:08:03.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:08:03.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:08:03.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:08:03.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:08:03.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:08:03.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:08:03.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:08:03.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:03.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:08:03.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:08:03.321 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:08:03.321 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:08:03.321 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:08:03.321 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:08:03.321 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:08:03.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:08:03.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:08:03.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:08:03.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2024-03-11T23:08:03.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:no valid command found; 5 closest matches: 2024-03-11T23:08:03.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status mirror/image_13 2024-03-11T23:08:03.429 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status mirror/image_14 2024-03-11T23:08:03.429 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status mirror/image_16 2024-03-11T23:08:03.429 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status mirror/image_17 2024-03-11T23:08:03.429 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status mirror/image_18 2024-03-11T23:08:03.429 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:08:03.432 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:08:03.432 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_1 2024-03-11T23:08:03.432 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:08:03.432 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:08:03.432 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:08:03.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:08:03.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:08:03.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:08:03.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:08:03.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:08:03.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:08:03.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:08:03.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:08:03.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:08:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:08:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:08:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:08:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:08:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:08:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:08:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:08:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:08:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:08:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:08:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:08:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:08:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:08:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:08:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:08:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:08:03.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:08:03.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:08:03.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:08:03.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_1 2024-03-11T23:08:03.533 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:no valid command found; 3 closest matches: 2024-03-11T23:08:03.533 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status mirror/image_11 2024-03-11T23:08:03.534 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status mirror/image_12 2024-03-11T23:08:03.534 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status mirror/image_15 2024-03-11T23:08:03.534 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:08:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:08:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_1 2024-03-11T23:08:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T23:08:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:08:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:08:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T23:08:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T23:08:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:08:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:08:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:08:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T23:08:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T23:08:03.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:08:03.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:03.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T23:08:03.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:08:03.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T23:08:03.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T23:08:03.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:08:03.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:08:03.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T23:08:03.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T23:08:03.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:08:03.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:08:03.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:03.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:08:03.539 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T23:08:03.539 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:08:03.539 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:08:03.539 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T23:08:03.539 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T23:08:03.539 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T23:08:03.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:08:03.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:08:03.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:08:03.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_1 2024-03-11T23:08:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:no valid command found; 2 closest matches: 2024-03-11T23:08:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status mirror/image_10 2024-03-11T23:08:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status mirror/image_19 2024-03-11T23:08:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:08:03.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:08:03.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_1 2024-03-11T23:08:03.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:3 2024-03-11T23:08:03.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:08:03.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:08:03.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-03-11T23:08:03.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:3 2024-03-11T23:08:03.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:08:03.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:08:03.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:08:03.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=3 2024-03-11T23:08:03.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 3 = cluster1:3 ']' 2024-03-11T23:08:03.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:08:03.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=3 2024-03-11T23:08:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:08:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-03-11T23:08:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:3 2024-03-11T23:08:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:08:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:08:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-03-11T23:08:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:3 2024-03-11T23:08:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:08:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:08:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:08:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 3 = cluster1:3 ']' 2024-03-11T23:08:03.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:08:03.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:08:03.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=3 2024-03-11T23:08:03.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=3 2024-03-11T23:08:03.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-03-11T23:08:03.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:08:03.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:08:03.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:08:03.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_1 2024-03-11T23:08:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:08:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:08:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:08:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:08:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:08:03.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:08:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:08:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:08:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2024-03-11T23:08:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:08:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T23:08:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:08:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:08:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_1 2024-03-11T23:08:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:08:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:03.752 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T23:08:03.752 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_1.status 2024-03-11T23:08:03.752 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_1.status 2024-03-11T23:08:03.752 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:08:03.752 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_1.status 2024-03-11T23:08:03.752 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_1.status 2024-03-11T23:08:03.752 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_1.status 2024-03-11T23:08:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:08:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_1.status 2024-03-11T23:08:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_1 2024-03-11T23:08:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T23:08:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_1 2024-03-11T23:08:04.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 86 2024-03-11T23:08:04.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_1 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_1.status 2024-03-11T23:08:04.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:04.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:04.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T23:08:04.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_1.status 2024-03-11T23:08:04.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_1 --format xml 2024-03-11T23:08:04.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:04.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:08:04.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=86 2024-03-11T23:08:04.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:08:04.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:08:04.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:08:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_1.status 2024-03-11T23:08:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T23:08:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_1.status 2024-03-11T23:08:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-03-11T23:08:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:04.819 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:08:04.821 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=35 2024-03-11T23:08:04.821 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 86 = 35 2024-03-11T23:08:04.821 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:08:04.822 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:08:05.223 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_1.status 2024-03-11T23:08:05.223 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:05.223 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:05.223 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T23:08:05.223 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_1.status 2024-03-11T23:08:05.223 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-03-11T23:08:05.223 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:08:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:08:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:08:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:08:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:08:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:08:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 86 = '' 2024-03-11T23:08:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:08:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:08:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_1.status 2024-03-11T23:08:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T23:08:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_1.status 2024-03-11T23:08:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-03-11T23:08:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:06.129 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:08:06.130 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:08:06.131 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:08:06.131 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:08:06.131 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:08:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:08:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 86 = '' 2024-03-11T23:08:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:08:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:08:07.733 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_1.status 2024-03-11T23:08:07.733 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:07.733 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:07.733 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T23:08:07.733 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_1.status 2024-03-11T23:08:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-03-11T23:08:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:08:07.786 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=86 2024-03-11T23:08:07.786 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 86 = 86 2024-03-11T23:08:07.786 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:08:07.786 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2024-03-11T23:08:07.786 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:07.786 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:07.786 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T23:08:07.786 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:08:07.786 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:08:07.786 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:08:07.786 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:08:08.787 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2024-03-11T23:08:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_1': 2024-03-11T23:08:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:08:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:08:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:08:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 10b31cf4e86a 2024-03-11T23:08:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.10b31cf4e86a 2024-03-11T23:08:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:08:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:08:08.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:08:08.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:08:08.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 22:59:12 2024 2024-03-11T23:08:08.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 22:59:12 2024 2024-03-11T23:08:08.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 22:59:12 2024 2024-03-11T23:08:08.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:08:08.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:08:08.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:08:08.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 7380f9c0-c386-40d3-8842-2968a362a7ba 2024-03-11T23:08:08.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:08:08.839 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:08:08.839 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_1 snap 2024-03-11T23:08:08.839 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:08.839 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T23:08:08.839 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:08:08.839 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:08:08.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_1.export 2024-03-11T23:08:08.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_1.export 2024-03-11T23:08:08.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_1.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_1.export 2024-03-11T23:08:08.840 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_1.export 2024-03-11T23:08:09.030 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:08:09.038 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_1.export 2024-03-11T23:08:09.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:08:09.243 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_1.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_1.export 2024-03-11T23:08:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_1.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_1.export 2024-03-11T23:08:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:08:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:08:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_2 2024-03-11T23:08:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_2 snap 2024-03-11T23:08:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:09.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T23:08:09.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:08:09.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:08:10.322 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2024-03-11T23:08:10.322 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:10.322 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:10.322 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T23:08:10.322 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2024-03-11T23:08:10.322 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:10.322 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:10.322 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T23:08:10.322 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:08:10.322 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:08:10.322 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:08:10.323 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:08:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2024-03-11T23:08:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T23:08:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:08:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:08:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:08:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2024-03-11T23:08:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:08:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:08:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:08:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:08:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:08:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:08:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:08:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:08:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:08:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:08:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:08:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2024-03-11T23:08:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:08:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:08:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:08:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:08:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:08:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:08:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:08:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:08:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:08:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:08:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:08:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:08:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:08:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:08:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:08:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:08:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:08:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:08:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:08:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:08:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:08:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:08:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:08:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:08:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:08:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:08:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:08:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:08:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:08:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:08:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:08:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2024-03-11T23:08:11.438 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:no valid command found; 3 closest matches: 2024-03-11T23:08:11.438 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status mirror/image_20 2024-03-11T23:08:11.438 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status mirror/image_26 2024-03-11T23:08:11.438 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status mirror/image_28 2024-03-11T23:08:11.438 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:08:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:08:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2024-03-11T23:08:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:08:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:08:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:08:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:08:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:08:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:08:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:08:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:08:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:08:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:08:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:08:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:08:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:08:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:08:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:08:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:08:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:08:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:08:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:08:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:08:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:08:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:08:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:08:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:08:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:08:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:08:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:08:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:08:11.460 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:08:11.460 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:08:11.460 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:08:11.460 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2024-03-11T23:08:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:no valid command found; 2 closest matches: 2024-03-11T23:08:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status mirror/image_22 2024-03-11T23:08:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status mirror/image_24 2024-03-11T23:08:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:08:11.546 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:08:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_2 2024-03-11T23:08:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T23:08:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:08:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:08:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T23:08:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T23:08:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:08:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:08:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:08:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T23:08:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T23:08:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:08:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T23:08:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:08:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T23:08:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T23:08:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:08:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:08:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T23:08:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T23:08:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:08:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:08:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:08:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T23:08:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:08:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:08:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T23:08:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T23:08:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T23:08:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:08:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:08:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:08:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_2 2024-03-11T23:08:11.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_2'] 2024-03-11T23:08:11.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:08:11.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:08:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:08:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_2 2024-03-11T23:08:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:3 2024-03-11T23:08:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:08:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:08:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-03-11T23:08:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:3 2024-03-11T23:08:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:08:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:08:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:08:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=3 2024-03-11T23:08:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 3 = cluster1:3 ']' 2024-03-11T23:08:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:08:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=3 2024-03-11T23:08:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:08:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-03-11T23:08:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:3 2024-03-11T23:08:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:08:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:08:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-03-11T23:08:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:3 2024-03-11T23:08:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:08:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:08:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:08:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 3 = cluster1:3 ']' 2024-03-11T23:08:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:08:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:08:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=3 2024-03-11T23:08:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=3 2024-03-11T23:08:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-03-11T23:08:11.669 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:08:11.670 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:08:11.670 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:08:11.670 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_2 2024-03-11T23:08:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:08:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:08:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:08:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:08:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:08:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:08:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:08:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:08:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2024-03-11T23:08:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:08:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T23:08:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:08:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:08:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_2 2024-03-11T23:08:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:08:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T23:08:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_2.status 2024-03-11T23:08:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_2.status 2024-03-11T23:08:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:08:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_2.status 2024-03-11T23:08:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_2.status 2024-03-11T23:08:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_2.status 2024-03-11T23:08:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:08:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_2.status 2024-03-11T23:08:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_2 2024-03-11T23:08:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T23:08:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_2 2024-03-11T23:08:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 88 2024-03-11T23:08:12.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_2 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_2.status 2024-03-11T23:08:12.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:12.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:12.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T23:08:12.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_2.status 2024-03-11T23:08:12.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_2 --format xml 2024-03-11T23:08:12.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:12.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:08:12.383 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=88 2024-03-11T23:08:12.383 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:08:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:08:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:08:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_2.status 2024-03-11T23:08:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T23:08:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_2.status 2024-03-11T23:08:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-03-11T23:08:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:12.630 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:08:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=36 2024-03-11T23:08:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 88 = 36 2024-03-11T23:08:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:08:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:08:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_2.status 2024-03-11T23:08:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T23:08:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_2.status 2024-03-11T23:08:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-03-11T23:08:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:13.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:08:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:08:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:08:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:08:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:08:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:08:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 88 = '' 2024-03-11T23:08:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:08:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:08:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_2.status 2024-03-11T23:08:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T23:08:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_2.status 2024-03-11T23:08:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-03-11T23:08:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:13.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:08:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:08:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:08:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:08:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:08:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:08:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 88 = '' 2024-03-11T23:08:13.957 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:08:13.957 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:08:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_2.status 2024-03-11T23:08:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T23:08:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_2.status 2024-03-11T23:08:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-03-11T23:08:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:08:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=88 2024-03-11T23:08:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 88 = 88 2024-03-11T23:08:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:08:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2024-03-11T23:08:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T23:08:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:08:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:08:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:08:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:08:16.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2024-03-11T23:08:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_2': 2024-03-11T23:08:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:08:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:08:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:08:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 11a7df61eacb 2024-03-11T23:08:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.11a7df61eacb 2024-03-11T23:08:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:08:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:08:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:08:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:08:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 22:59:24 2024 2024-03-11T23:08:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 22:59:24 2024 2024-03-11T23:08:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 22:59:24 2024 2024-03-11T23:08:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:08:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:08:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:08:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 0db5ab54-08c1-4a9b-b8f1-d2ac90c55bee 2024-03-11T23:08:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:08:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:08:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_2 snap 2024-03-11T23:08:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T23:08:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:08:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:08:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_2.export 2024-03-11T23:08:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_2.export 2024-03-11T23:08:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_2.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_2.export 2024-03-11T23:08:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_2.export 2024-03-11T23:08:16.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:08:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_2.export 2024-03-11T23:08:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:08:17.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_2.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_2.export 2024-03-11T23:08:17.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_2.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_2.export 2024-03-11T23:08:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:08:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:08:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_3 2024-03-11T23:08:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_3 snap 2024-03-11T23:08:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T23:08:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:08:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:08:17.331 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2024-03-11T23:08:17.331 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:17.331 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:17.331 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T23:08:17.331 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2024-03-11T23:08:17.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:17.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:17.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T23:08:17.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:08:17.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:08:17.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:08:17.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:08:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2024-03-11T23:08:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T23:08:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:08:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:08:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:08:18.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2024-03-11T23:08:18.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:08:18.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:08:18.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:08:18.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:08:18.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:08:18.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:08:18.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:08:18.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:08:18.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:08:18.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:08:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:08:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2024-03-11T23:08:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:08:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:08:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:08:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:08:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:08:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:08:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:08:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:08:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:08:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:08:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:08:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:08:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:08:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:08:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:08:18.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:08:18.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:08:18.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:08:18.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:08:18.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:08:18.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:08:18.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:18.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:08:18.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:08:18.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:08:18.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:08:18.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:08:18.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:08:18.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:08:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:08:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:08:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:08:18.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2024-03-11T23:08:18.447 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:no valid command found; 1 closest matches: 2024-03-11T23:08:18.447 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status mirror/image_36 2024-03-11T23:08:18.447 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:08:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:08:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_3 2024-03-11T23:08:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:08:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:08:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:08:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:08:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:08:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:08:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:08:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:08:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:08:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:08:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:08:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:08:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:08:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:08:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:08:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:08:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:08:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:08:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:08:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:08:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:08:18.453 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:18.453 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:08:18.453 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:08:18.453 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:08:18.453 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:08:18.453 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:08:18.453 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:08:18.453 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:08:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:08:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:08:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:08:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_3 2024-03-11T23:08:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:08:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:08:18.562 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:08:18.562 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:08:18.562 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:08:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:08:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:08:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:08:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2024-03-11T23:08:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:08:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T23:08:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:08:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:08:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_3 2024-03-11T23:08:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:08:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T23:08:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_3.status 2024-03-11T23:08:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_3.status 2024-03-11T23:08:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:08:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_3.status 2024-03-11T23:08:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_3.status 2024-03-11T23:08:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_3.status 2024-03-11T23:08:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:08:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_3.status 2024-03-11T23:08:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_3 2024-03-11T23:08:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T23:08:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_3 2024-03-11T23:08:19.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 90 2024-03-11T23:08:19.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_3 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_3.status 2024-03-11T23:08:19.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:19.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:19.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T23:08:19.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_3.status 2024-03-11T23:08:19.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_3 --format xml 2024-03-11T23:08:19.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:19.409 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:08:19.410 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=90 2024-03-11T23:08:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:08:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:08:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:08:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_3.status 2024-03-11T23:08:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T23:08:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_3.status 2024-03-11T23:08:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-03-11T23:08:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:08:19.662 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=37 2024-03-11T23:08:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 90 = 37 2024-03-11T23:08:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:08:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:08:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_3.status 2024-03-11T23:08:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T23:08:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_3.status 2024-03-11T23:08:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-03-11T23:08:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:08:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:08:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:08:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:08:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:08:20.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:08:20.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 90 = '' 2024-03-11T23:08:20.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:08:20.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:08:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_3.status 2024-03-11T23:08:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T23:08:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_3.status 2024-03-11T23:08:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-03-11T23:08:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:08:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:08:21.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:08:21.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:08:21.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:08:21.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:08:21.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 90 = '' 2024-03-11T23:08:21.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:08:21.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:08:22.651 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_3.status 2024-03-11T23:08:22.651 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:22.651 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:22.651 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T23:08:22.651 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_3.status 2024-03-11T23:08:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-03-11T23:08:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:22.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:08:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=90 2024-03-11T23:08:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 90 = 90 2024-03-11T23:08:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:08:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2024-03-11T23:08:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T23:08:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:08:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:08:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:08:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:08:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2024-03-11T23:08:23.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_3': 2024-03-11T23:08:23.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:08:23.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:08:23.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:08:23.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 11ffedb6de10 2024-03-11T23:08:23.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.11ffedb6de10 2024-03-11T23:08:23.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:08:23.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:08:23.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:08:23.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:08:23.754 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 22:59:36 2024 2024-03-11T23:08:23.754 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 22:59:36 2024 2024-03-11T23:08:23.754 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 22:59:36 2024 2024-03-11T23:08:23.754 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:08:23.754 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:08:23.754 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:08:23.754 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 89903746-8b7b-4dea-b492-96c6f9aebf71 2024-03-11T23:08:23.754 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:08:23.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:08:23.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_3 snap 2024-03-11T23:08:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T23:08:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:08:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:08:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_3.export 2024-03-11T23:08:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_3.export 2024-03-11T23:08:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_3.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_3.export 2024-03-11T23:08:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_3.export 2024-03-11T23:08:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:08:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_3.export 2024-03-11T23:08:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:08:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_3.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_3.export 2024-03-11T23:08:24.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_3.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_3.export 2024-03-11T23:08:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:08:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:08:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_4 2024-03-11T23:08:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_4 snap 2024-03-11T23:08:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T23:08:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:08:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:08:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2024-03-11T23:08:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T23:08:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2024-03-11T23:08:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T23:08:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:08:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:08:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:08:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:08:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2024-03-11T23:08:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T23:08:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:08:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:08:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:08:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2024-03-11T23:08:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:08:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:08:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:08:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:08:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:08:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:08:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:08:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:08:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:08:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:08:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:08:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2024-03-11T23:08:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:08:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:08:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:08:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:08:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:08:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:08:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:08:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:08:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:08:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:08:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:08:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:08:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:08:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:08:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:08:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:08:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:08:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:08:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:08:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:08:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:08:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:08:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:08:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:08:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:08:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:08:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:08:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:08:25.906 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9391714178581402) > weight (0.25) 2024-03-11T23:08:25.907 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.3834340730965756) > weight (0.25) 2024-03-11T23:08:25.907 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-03-11T23:08:25.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-03-11T23:08:25.907 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-03-11T23:08:25.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi132.stderr:2024-03-11T23:08:25.904+0000 7ff432d02640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 50571) UID: 1000 2024-03-11T23:08:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:08:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:08:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:08:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2024-03-11T23:08:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:no valid command found; 2 closest matches: 2024-03-11T23:08:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status mirror/image_43 2024-03-11T23:08:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status mirror/image_47 2024-03-11T23:08:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:08:26.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:08:26.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2024-03-11T23:08:26.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:08:26.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:08:26.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:08:26.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:08:26.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:08:26.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:08:26.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:08:26.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:08:26.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:08:26.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:08:26.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:08:26.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:26.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:08:26.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:08:26.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:08:26.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:08:26.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:08:26.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:08:26.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:08:26.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:08:26.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:08:26.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:08:26.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:26.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:08:26.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:08:26.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:08:26.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:08:26.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:08:26.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:08:26.026 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:08:26.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:08:26.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:08:26.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:08:26.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2024-03-11T23:08:26.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:08:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:08:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:08:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:08:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:08:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:08:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:08:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:08:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2024-03-11T23:08:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:08:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T23:08:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:08:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:08:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_4 2024-03-11T23:08:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:08:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T23:08:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_4.status 2024-03-11T23:08:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_4.status 2024-03-11T23:08:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:08:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_4.status 2024-03-11T23:08:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_4.status 2024-03-11T23:08:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_4.status 2024-03-11T23:08:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:08:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_4.status 2024-03-11T23:08:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_4 2024-03-11T23:08:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T23:08:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_4 2024-03-11T23:08:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 92 2024-03-11T23:08:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_4 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_4.status 2024-03-11T23:08:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T23:08:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_4.status 2024-03-11T23:08:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_4 --format xml 2024-03-11T23:08:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:26.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:08:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=92 2024-03-11T23:08:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:08:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:08:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:08:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_4.status 2024-03-11T23:08:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T23:08:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_4.status 2024-03-11T23:08:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-03-11T23:08:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:08:27.215 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=38 2024-03-11T23:08:27.215 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 92 = 38 2024-03-11T23:08:27.215 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:08:27.215 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:08:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_4.status 2024-03-11T23:08:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T23:08:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_4.status 2024-03-11T23:08:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-03-11T23:08:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:08:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=38 2024-03-11T23:08:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 92 = 38 2024-03-11T23:08:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:08:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:08:28.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_4.status 2024-03-11T23:08:28.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:28.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:28.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T23:08:28.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_4.status 2024-03-11T23:08:28.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-03-11T23:08:28.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:28.522 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:08:28.522 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:08:28.523 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:08:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:08:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:08:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:08:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 92 = '' 2024-03-11T23:08:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:08:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:08:29.908 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-03-11T23:08:29.909 INFO:teuthology.orchestra.run:waiting for 600 2024-03-11T23:08:29.909 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-03-11T23:08:29.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-03-11T23:08:29.909 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-03-11T23:08:29.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-03-11T23:08:29.912 INFO:tasks.rbd_mirror_thrash:waiting for 69 secs before thrashing 2024-03-11T23:08:30.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_4.status 2024-03-11T23:08:30.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:30.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:30.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T23:08:30.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_4.status 2024-03-11T23:08:30.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-03-11T23:08:30.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:30.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:08:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=92 2024-03-11T23:08:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 92 = 92 2024-03-11T23:08:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:08:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2024-03-11T23:08:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T23:08:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:08:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:08:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:08:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:08:31.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2024-03-11T23:08:31.217 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_4': 2024-03-11T23:08:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:08:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:08:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:08:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 1260b2ceddc2 2024-03-11T23:08:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.1260b2ceddc2 2024-03-11T23:08:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:08:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:08:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:08:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:08:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 22:59:48 2024 2024-03-11T23:08:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 22:59:48 2024 2024-03-11T23:08:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 22:59:48 2024 2024-03-11T23:08:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:08:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:08:31.219 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:08:31.219 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: fc8576a3-dbba-4c16-95dd-f75d6193b45b 2024-03-11T23:08:31.219 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:08:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:08:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_4 snap 2024-03-11T23:08:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T23:08:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:08:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:08:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_4.export 2024-03-11T23:08:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_4.export 2024-03-11T23:08:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_4.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_4.export 2024-03-11T23:08:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_4.export 2024-03-11T23:08:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:08:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_4.export 2024-03-11T23:08:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:08:31.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_4.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_4.export 2024-03-11T23:08:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_4.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_4.export 2024-03-11T23:08:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:08:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:08:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_5 2024-03-11T23:08:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_5 snap 2024-03-11T23:08:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T23:08:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:08:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:08:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2024-03-11T23:08:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T23:08:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2024-03-11T23:08:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T23:08:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:08:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:08:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:08:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:08:33.307 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2024-03-11T23:08:33.307 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:33.307 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T23:08:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:08:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:08:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:08:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2024-03-11T23:08:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:08:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:08:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:08:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:08:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:08:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:08:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:08:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:08:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:08:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:08:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:08:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2024-03-11T23:08:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:08:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:08:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:08:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:08:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:08:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:08:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:08:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:08:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:08:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:08:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:08:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:08:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:08:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:08:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:08:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:08:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:08:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:08:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:08:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:08:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:08:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:08:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:08:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:08:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:08:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:08:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:08:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:08:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:08:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:08:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:08:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2024-03-11T23:08:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-03-11T23:08:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:08:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:08:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:08:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2024-03-11T23:08:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:08:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:08:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:08:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:08:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:08:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:08:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:08:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:08:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:08:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:08:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:08:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:08:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:08:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:08:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:08:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:08:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:08:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:08:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:08:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:08:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:08:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:08:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:08:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:08:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:08:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:08:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:08:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:08:33.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:08:33.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:08:33.444 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:08:33.444 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2024-03-11T23:08:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:no valid command found; 1 closest matches: 2024-03-11T23:08:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status mirror/image_50 2024-03-11T23:08:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:08:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:08:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2024-03-11T23:08:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T23:08:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:08:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:08:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T23:08:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T23:08:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:08:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:08:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:08:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T23:08:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T23:08:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:08:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T23:08:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:08:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T23:08:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T23:08:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:08:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:08:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T23:08:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T23:08:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:08:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:08:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:08:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T23:08:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:08:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:08:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T23:08:33.536 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T23:08:33.536 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T23:08:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:08:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:08:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:08:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2024-03-11T23:08:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:08:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:08:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:08:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:08:33.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:08:33.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:08:33.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:08:33.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:08:33.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2024-03-11T23:08:33.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:08:33.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:33.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:33.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T23:08:33.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:08:33.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:08:33.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_5 2024-03-11T23:08:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:08:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T23:08:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_5.status 2024-03-11T23:08:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_5.status 2024-03-11T23:08:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:08:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_5.status 2024-03-11T23:08:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_5.status 2024-03-11T23:08:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_5.status 2024-03-11T23:08:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:08:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_5.status 2024-03-11T23:08:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_5 2024-03-11T23:08:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T23:08:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_5 2024-03-11T23:08:34.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 94 2024-03-11T23:08:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_5 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_5.status 2024-03-11T23:08:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T23:08:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_5.status 2024-03-11T23:08:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_5 --format xml 2024-03-11T23:08:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:08:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=94 2024-03-11T23:08:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:08:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:08:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:08:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_5.status 2024-03-11T23:08:34.566 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:34.566 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:34.566 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T23:08:34.566 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_5.status 2024-03-11T23:08:34.566 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-03-11T23:08:34.566 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:34.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:08:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=39 2024-03-11T23:08:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 94 = 39 2024-03-11T23:08:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:08:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:08:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_5.status 2024-03-11T23:08:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T23:08:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_5.status 2024-03-11T23:08:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-03-11T23:08:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:08:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:08:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:08:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:08:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:08:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:08:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 94 = '' 2024-03-11T23:08:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:08:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:08:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_5.status 2024-03-11T23:08:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T23:08:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_5.status 2024-03-11T23:08:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-03-11T23:08:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:08:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:08:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:08:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:08:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:08:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:08:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 94 = '' 2024-03-11T23:08:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:08:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:08:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_5.status 2024-03-11T23:08:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T23:08:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_5.status 2024-03-11T23:08:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-03-11T23:08:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:08:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=94 2024-03-11T23:08:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 94 = 94 2024-03-11T23:08:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:08:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2024-03-11T23:08:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T23:08:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:08:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:08:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:08:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:08:38.567 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2024-03-11T23:08:38.613 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_5': 2024-03-11T23:08:38.613 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:08:38.613 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:08:38.613 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:08:38.613 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 10b3b82fea9d 2024-03-11T23:08:38.613 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.10b3b82fea9d 2024-03-11T23:08:38.613 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:08:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:08:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:08:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:08:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 22:59:58 2024 2024-03-11T23:08:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 22:59:58 2024 2024-03-11T23:08:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 22:59:58 2024 2024-03-11T23:08:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:08:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:08:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:08:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 6e7c5b9b-f490-4f68-8ae3-de136df2e981 2024-03-11T23:08:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:08:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:08:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_5 snap 2024-03-11T23:08:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T23:08:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:08:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:08:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_5.export 2024-03-11T23:08:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_5.export 2024-03-11T23:08:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_5.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_5.export 2024-03-11T23:08:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_5.export 2024-03-11T23:08:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:08:38.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_5.export 2024-03-11T23:08:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:08:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_5.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_5.export 2024-03-11T23:08:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_5.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_5.export 2024-03-11T23:08:39.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:08:39.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:08:39.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_6 2024-03-11T23:08:39.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_6 snap 2024-03-11T23:08:39.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:39.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:39.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:08:39.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:08:39.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:08:39.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2024-03-11T23:08:39.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:39.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:39.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:08:39.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2024-03-11T23:08:39.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:39.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:39.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:08:39.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:08:39.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:08:39.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:08:39.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:08:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2024-03-11T23:08:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:08:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:08:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:08:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:08:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2024-03-11T23:08:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:08:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:08:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:08:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:08:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:08:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:08:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:08:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:08:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:08:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:08:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:08:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2024-03-11T23:08:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:08:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:08:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:08:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:08:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:08:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:08:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:08:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:08:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:08:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:08:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:08:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:08:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:08:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:08:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:08:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:08:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:08:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:08:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:08:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:08:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:08:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:08:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:08:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:08:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:08:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:08:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:08:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:08:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:08:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:08:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:08:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2024-03-11T23:08:40.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:08:40.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:08:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:08:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:08:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:08:40.440 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:08:40.440 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:08:40.440 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:08:40.440 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2024-03-11T23:08:40.440 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:08:40.440 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:08:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:08:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:08:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_6 2024-03-11T23:08:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:08:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:08:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_6.status 2024-03-11T23:08:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_6.status 2024-03-11T23:08:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:08:40.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_6.status 2024-03-11T23:08:40.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_6.status 2024-03-11T23:08:40.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_6.status 2024-03-11T23:08:40.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:08:40.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_6.status 2024-03-11T23:08:40.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_6 2024-03-11T23:08:40.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:40.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:40.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:08:40.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_6 2024-03-11T23:08:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 96 2024-03-11T23:08:41.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_6 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_6.status 2024-03-11T23:08:41.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:08:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_6.status 2024-03-11T23:08:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_6 --format xml 2024-03-11T23:08:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:08:41.375 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=96 2024-03-11T23:08:41.375 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:08:41.375 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:08:41.375 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:08:41.576 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_6.status 2024-03-11T23:08:41.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:41.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:41.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:08:41.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_6.status 2024-03-11T23:08:41.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-03-11T23:08:41.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:08:41.623 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=40 2024-03-11T23:08:41.623 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 96 = 40 2024-03-11T23:08:41.623 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:08:41.623 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:08:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_6.status 2024-03-11T23:08:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:08:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_6.status 2024-03-11T23:08:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-03-11T23:08:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:08:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:08:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:08:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:08:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:08:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:08:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 96 = '' 2024-03-11T23:08:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:08:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:08:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_6.status 2024-03-11T23:08:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:08:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_6.status 2024-03-11T23:08:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-03-11T23:08:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:42.928 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:08:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:08:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:08:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:08:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:08:42.931 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:08:42.931 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 96 = '' 2024-03-11T23:08:42.931 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:08:42.931 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:08:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_6.status 2024-03-11T23:08:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:08:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_6.status 2024-03-11T23:08:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-03-11T23:08:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:44.586 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:08:44.588 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=96 2024-03-11T23:08:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 96 = 96 2024-03-11T23:08:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:08:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2024-03-11T23:08:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:08:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:08:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:08:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:08:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:08:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2024-03-11T23:08:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_6': 2024-03-11T23:08:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:08:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:08:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:08:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 127f6cfa1423 2024-03-11T23:08:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.127f6cfa1423 2024-03-11T23:08:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:08:45.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:08:45.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:08:45.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:08:45.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:00:39 2024 2024-03-11T23:08:45.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:00:39 2024 2024-03-11T23:08:45.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:00:39 2024 2024-03-11T23:08:45.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:08:45.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:08:45.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:08:45.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: fdce66ca-291f-45dc-b094-a088660b842f 2024-03-11T23:08:45.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:08:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:08:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_6 snap 2024-03-11T23:08:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:08:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:08:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:08:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_6.export 2024-03-11T23:08:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_6.export 2024-03-11T23:08:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_6.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_6.export 2024-03-11T23:08:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_6.export 2024-03-11T23:08:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:08:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_6.export 2024-03-11T23:08:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:08:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_6.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_6.export 2024-03-11T23:08:46.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_6.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_6.export 2024-03-11T23:08:46.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:08:46.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:08:46.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_7 2024-03-11T23:08:46.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_7 snap 2024-03-11T23:08:46.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:46.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:46.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:08:46.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:08:46.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:08:46.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2024-03-11T23:08:46.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:46.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:46.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:08:46.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2024-03-11T23:08:46.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:46.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:46.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:08:46.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:08:46.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:08:46.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:08:46.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:08:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2024-03-11T23:08:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:47.331 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:08:47.331 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:08:47.331 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:08:47.331 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:08:47.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2024-03-11T23:08:47.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:08:47.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:08:47.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:08:47.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:08:47.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:08:47.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:08:47.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:08:47.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:08:47.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:08:47.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:08:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:08:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2024-03-11T23:08:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:08:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:08:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:08:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:08:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:08:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:08:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:08:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:08:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:08:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:08:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:08:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:08:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:08:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:08:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:08:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:08:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:08:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:08:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:08:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:08:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:08:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:08:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:08:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:08:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:08:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:08:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:08:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:08:47.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:08:47.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:08:47.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:08:47.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2024-03-11T23:08:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-03-11T23:08:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:08:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:08:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:08:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_7 2024-03-11T23:08:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:08:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:08:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:08:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:08:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:08:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:08:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:08:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:08:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:08:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:08:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:08:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:08:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:08:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:08:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:08:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:08:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:08:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:08:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:08:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:08:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:08:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:08:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:08:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:08:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:08:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:08:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:08:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:08:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:08:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:08:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:08:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_7 2024-03-11T23:08:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-03-11T23:08:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:08:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:08:47.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:08:47.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_7 2024-03-11T23:08:47.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T23:08:47.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:08:47.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:08:47.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T23:08:47.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T23:08:47.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:08:47.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:08:47.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:08:47.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T23:08:47.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T23:08:47.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:08:47.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:47.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T23:08:47.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:08:47.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T23:08:47.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T23:08:47.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:08:47.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:08:47.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T23:08:47.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T23:08:47.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:08:47.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:08:47.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:47.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:08:47.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T23:08:47.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:08:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:08:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T23:08:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T23:08:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T23:08:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:08:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:08:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:08:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_7 2024-03-11T23:08:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-03-11T23:08:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:08:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:08:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:08:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_7 2024-03-11T23:08:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:3 2024-03-11T23:08:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:08:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:08:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-03-11T23:08:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:3 2024-03-11T23:08:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:08:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:08:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:08:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=3 2024-03-11T23:08:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 3 = cluster1:3 ']' 2024-03-11T23:08:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:08:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=3 2024-03-11T23:08:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:08:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-03-11T23:08:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:3 2024-03-11T23:08:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:08:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:08:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-03-11T23:08:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:3 2024-03-11T23:08:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:08:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:08:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:08:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 3 = cluster1:3 ']' 2024-03-11T23:08:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:08:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:08:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=3 2024-03-11T23:08:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=3 2024-03-11T23:08:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-03-11T23:08:47.673 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:08:47.673 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:08:47.673 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:08:47.673 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_7 2024-03-11T23:08:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:08:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:08:47.765 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:08:47.765 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:08:47.765 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:08:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:08:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:08:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:08:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2024-03-11T23:08:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:08:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:47.768 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:08:47.768 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:08:47.768 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:08:47.768 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_7 2024-03-11T23:08:47.768 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:08:47.768 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:47.768 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:47.768 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:08:47.768 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_7.status 2024-03-11T23:08:47.768 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_7.status 2024-03-11T23:08:47.768 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:08:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_7.status 2024-03-11T23:08:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_7.status 2024-03-11T23:08:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_7.status 2024-03-11T23:08:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:08:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_7.status 2024-03-11T23:08:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_7 2024-03-11T23:08:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:08:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_7 2024-03-11T23:08:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 98 2024-03-11T23:08:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_7 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_7.status 2024-03-11T23:08:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:08:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_7.status 2024-03-11T23:08:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_7 --format xml 2024-03-11T23:08:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:08:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=98 2024-03-11T23:08:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:08:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:08:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:08:48.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_7.status 2024-03-11T23:08:48.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:08:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_7.status 2024-03-11T23:08:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-03-11T23:08:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:48.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:08:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=41 2024-03-11T23:08:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 98 = 41 2024-03-11T23:08:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:08:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:08:49.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_7.status 2024-03-11T23:08:49.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:49.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:49.047 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:08:49.047 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_7.status 2024-03-11T23:08:49.047 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-03-11T23:08:49.047 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:08:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:08:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:08:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:08:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:08:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:08:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 98 = '' 2024-03-11T23:08:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:08:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:08:49.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_7.status 2024-03-11T23:08:49.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:49.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:49.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:08:49.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_7.status 2024-03-11T23:08:49.899 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-03-11T23:08:49.899 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:08:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:08:49.945 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:08:49.945 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:08:49.945 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:08:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:08:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 98 = '' 2024-03-11T23:08:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:08:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:08:51.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_7.status 2024-03-11T23:08:51.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:51.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:51.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:08:51.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_7.status 2024-03-11T23:08:51.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-03-11T23:08:51.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:51.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:08:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=98 2024-03-11T23:08:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 98 = 98 2024-03-11T23:08:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:08:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2024-03-11T23:08:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:08:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:08:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:08:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:08:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:08:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2024-03-11T23:08:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_7': 2024-03-11T23:08:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:08:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:08:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:08:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 127f1c4d02a8 2024-03-11T23:08:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.127f1c4d02a8 2024-03-11T23:08:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:08:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:08:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:08:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:08:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:00:21 2024 2024-03-11T23:08:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:00:21 2024 2024-03-11T23:08:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:00:21 2024 2024-03-11T23:08:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:08:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:08:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:08:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: e7d22bec-3c5d-4227-9c8e-5fb052d5be91 2024-03-11T23:08:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:08:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:08:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_7 snap 2024-03-11T23:08:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:08:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:08:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:08:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_7.export 2024-03-11T23:08:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_7.export 2024-03-11T23:08:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_7.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_7.export 2024-03-11T23:08:52.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_7.export 2024-03-11T23:08:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:08:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_7.export 2024-03-11T23:08:53.097 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:08:53.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_7.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_7.export 2024-03-11T23:08:53.165 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_7.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_7.export 2024-03-11T23:08:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:08:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:08:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_8 2024-03-11T23:08:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_8 snap 2024-03-11T23:08:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_8 2024-03-11T23:08:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:08:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:08:53.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2024-03-11T23:08:53.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:53.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:53.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_8 2024-03-11T23:08:53.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2024-03-11T23:08:53.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:53.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:53.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_8 2024-03-11T23:08:53.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:08:53.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:08:53.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:08:53.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:08:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2024-03-11T23:08:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_8 2024-03-11T23:08:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:08:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:08:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:08:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2024-03-11T23:08:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:08:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:08:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:08:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:08:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:08:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:08:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:08:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:08:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:08:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:08:54.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:08:54.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2024-03-11T23:08:54.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:08:54.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:08:54.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:08:54.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:08:54.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:08:54.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:08:54.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:08:54.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:08:54.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:08:54.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:08:54.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:08:54.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:54.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:08:54.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:08:54.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:08:54.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:08:54.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:08:54.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:08:54.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:08:54.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:08:54.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:08:54.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:08:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:08:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:08:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:08:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:08:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:08:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:08:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:08:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:08:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:08:54.362 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:08:54.362 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2024-03-11T23:08:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-03-11T23:08:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:08:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:08:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:08:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_8 2024-03-11T23:08:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:08:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:08:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:08:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:08:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:08:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:08:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:08:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:08:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:08:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:08:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:08:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:08:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:08:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:08:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:08:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:08:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:08:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:08:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:08:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:08:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:08:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:08:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:08:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:08:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:08:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:08:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:08:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:08:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:08:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:08:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:08:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_8 2024-03-11T23:08:54.550 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-03-11T23:08:54.550 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:08:54.551 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:08:54.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:08:54.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_8 2024-03-11T23:08:54.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T23:08:54.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:08:54.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:08:54.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T23:08:54.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T23:08:54.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:08:54.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:08:54.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:08:54.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T23:08:54.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T23:08:54.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:08:54.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T23:08:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:08:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T23:08:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T23:08:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:08:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:08:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T23:08:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T23:08:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:08:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:08:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:08:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:08:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T23:08:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:08:54.556 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:08:54.556 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T23:08:54.556 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T23:08:54.556 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T23:08:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:08:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:08:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:08:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_8 2024-03-11T23:08:54.661 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:08:54.661 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:08:54.661 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:08:54.661 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:08:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:08:54.663 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:08:54.663 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:08:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:08:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2024-03-11T23:08:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:08:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_8 2024-03-11T23:08:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:08:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:08:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_8 2024-03-11T23:08:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:08:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_8 2024-03-11T23:08:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_8.status 2024-03-11T23:08:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_8.status 2024-03-11T23:08:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:08:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_8.status 2024-03-11T23:08:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_8.status 2024-03-11T23:08:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_8.status 2024-03-11T23:08:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:08:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_8.status 2024-03-11T23:08:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_8 2024-03-11T23:08:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_8 2024-03-11T23:08:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_8 2024-03-11T23:08:55.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 100 2024-03-11T23:08:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_8 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_8.status 2024-03-11T23:08:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:08:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_8 2024-03-11T23:08:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_8.status 2024-03-11T23:08:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_8 --format xml 2024-03-11T23:08:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:08:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=100 2024-03-11T23:08:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:08:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:08:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:08:55.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_8.status 2024-03-11T23:08:55.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:55.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:55.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_8 2024-03-11T23:08:55.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_8.status 2024-03-11T23:08:55.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-03-11T23:08:55.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:55.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:08:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=42 2024-03-11T23:08:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 100 = 42 2024-03-11T23:08:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:08:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:08:56.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_8.status 2024-03-11T23:08:56.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:56.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:56.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_8 2024-03-11T23:08:56.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_8.status 2024-03-11T23:08:56.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-03-11T23:08:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:08:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:08:56.095 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:08:56.095 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:08:56.095 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:08:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:08:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 100 = '' 2024-03-11T23:08:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:08:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:08:56.817 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7357186827960921) > weight (0.25) 2024-03-11T23:08:56.817 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7682291812355982) > weight (0.25) 2024-03-11T23:08:56.818 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5087153219812778) > weight (0.25) 2024-03-11T23:08:56.818 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-03-11T23:08:56.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-03-11T23:08:56.818 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-03-11T23:08:56.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi132.stderr:2024-03-11T23:08:56.815+0000 7fb528330640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 40027) UID: 1000 2024-03-11T23:08:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_8.status 2024-03-11T23:08:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_8 2024-03-11T23:08:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_8.status 2024-03-11T23:08:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-03-11T23:08:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:08:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:08:56.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:08:56.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:08:56.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:08:56.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:08:56.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 100 = '' 2024-03-11T23:08:56.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:08:56.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:08:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_8.status 2024-03-11T23:08:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_8 2024-03-11T23:08:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_8.status 2024-03-11T23:08:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-03-11T23:08:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:08:58.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:08:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=100 2024-03-11T23:08:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 100 = 100 2024-03-11T23:08:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:08:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2024-03-11T23:08:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:08:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_8 2024-03-11T23:08:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:08:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:08:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:08:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:08:58.819 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-03-11T23:08:58.819 INFO:teuthology.orchestra.run:waiting for 600 2024-03-11T23:08:58.820 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-03-11T23:08:58.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-03-11T23:08:58.820 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-03-11T23:08:58.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-03-11T23:08:58.823 INFO:tasks.rbd_mirror_thrash:waiting for 98 secs before thrashing 2024-03-11T23:08:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2024-03-11T23:08:59.670 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_8': 2024-03-11T23:08:59.670 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:08:59.670 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:08:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:08:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 11ffc9f90e42 2024-03-11T23:08:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.11ffc9f90e42 2024-03-11T23:08:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:08:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:08:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:08:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:08:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:00:35 2024 2024-03-11T23:08:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:00:35 2024 2024-03-11T23:08:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:00:35 2024 2024-03-11T23:08:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:08:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:08:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:08:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: ac93902e-f852-49af-ac1f-b13ea2b5269b 2024-03-11T23:08:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:08:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:08:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_8 snap 2024-03-11T23:08:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:08:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_8 2024-03-11T23:08:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:08:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:08:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_8.export 2024-03-11T23:08:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_8.export 2024-03-11T23:08:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_8.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_8.export 2024-03-11T23:08:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_8.export 2024-03-11T23:08:59.864 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:08:59.872 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_8.export 2024-03-11T23:09:00.046 INFO:tasks.ceph.cluster2.osd.2.smithi132.stderr:2024-03-11T23:09:00.040+0000 7f6376022640 -1 reset not still connected to 0x56507b4f1c70 2024-03-11T23:09:00.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:09:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_8.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_8.export 2024-03-11T23:09:00.178 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_8.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_8.export 2024-03-11T23:09:00.208 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:09:00.208 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:09:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_9 2024-03-11T23:09:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_9 snap 2024-03-11T23:09:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_9 2024-03-11T23:09:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:09:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:09:00.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2024-03-11T23:09:00.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:00.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:00.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_9 2024-03-11T23:09:00.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2024-03-11T23:09:00.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:00.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:00.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_9 2024-03-11T23:09:00.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:09:00.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:09:00.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:09:00.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:09:01.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2024-03-11T23:09:01.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:01.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:01.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_9 2024-03-11T23:09:01.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:09:01.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:09:01.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:09:01.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2024-03-11T23:09:01.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:09:01.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:09:01.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:09:01.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:09:01.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:09:01.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:09:01.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:09:01.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:09:01.359 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:09:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:09:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:09:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2024-03-11T23:09:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:09:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:09:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:09:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:09:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:09:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:09:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:09:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:09:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:09:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:09:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:09:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:09:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:09:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:09:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:09:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:09:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:09:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:09:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:09:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:09:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:09:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:09:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:09:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:09:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:09:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:09:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:09:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:09:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:09:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:09:01.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:09:01.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:09:01.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:09:01.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2024-03-11T23:09:01.470 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2024-03-11T23:09:01.470 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:09:01.470 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:09:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:09:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_9 2024-03-11T23:09:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:09:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:09:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:09:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:09:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:09:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:09:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:09:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:09:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:09:01.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:09:01.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:09:01.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:09:01.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:09:01.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:09:01.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:09:01.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:09:01.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:09:01.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:09:01.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:09:01.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:09:01.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:09:01.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:09:01.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:09:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:09:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:09:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:09:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:09:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:09:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:09:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:09:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:09:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:09:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:09:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_9 2024-03-11T23:09:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2024-03-11T23:09:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:09:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:09:01.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:09:01.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_9 2024-03-11T23:09:01.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T23:09:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:09:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:09:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T23:09:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T23:09:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:09:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:09:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:09:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T23:09:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T23:09:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:09:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:09:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T23:09:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:09:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T23:09:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T23:09:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:09:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:09:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T23:09:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T23:09:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:09:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:09:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:09:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:09:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T23:09:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:09:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:09:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T23:09:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T23:09:01.583 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T23:09:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:09:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:09:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:09:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_9 2024-03-11T23:09:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:09:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:09:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:09:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:09:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:09:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:09:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:09:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:09:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2024-03-11T23:09:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:09:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_9 2024-03-11T23:09:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:09:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:09:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_9 2024-03-11T23:09:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:09:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_9 2024-03-11T23:09:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_9.status 2024-03-11T23:09:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_9.status 2024-03-11T23:09:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:09:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_9.status 2024-03-11T23:09:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_9.status 2024-03-11T23:09:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_9.status 2024-03-11T23:09:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:09:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_9.status 2024-03-11T23:09:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_9 2024-03-11T23:09:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_9 2024-03-11T23:09:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_9 2024-03-11T23:09:02.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 102 2024-03-11T23:09:02.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_9 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_9.status 2024-03-11T23:09:02.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:02.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:02.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_9 2024-03-11T23:09:02.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_9.status 2024-03-11T23:09:02.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_9 --format xml 2024-03-11T23:09:02.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:02.410 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:09:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=102 2024-03-11T23:09:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:09:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:09:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:09:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_9.status 2024-03-11T23:09:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:02.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:02.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_9 2024-03-11T23:09:02.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_9.status 2024-03-11T23:09:02.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-03-11T23:09:02.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:09:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=43 2024-03-11T23:09:02.662 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 102 = 43 2024-03-11T23:09:02.662 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:09:02.662 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:09:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_9.status 2024-03-11T23:09:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_9 2024-03-11T23:09:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_9.status 2024-03-11T23:09:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-03-11T23:09:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:03.110 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:09:03.110 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:09:03.111 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:09:03.111 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:09:03.111 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:09:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:09:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 102 = '' 2024-03-11T23:09:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:09:03.113 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:09:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_9.status 2024-03-11T23:09:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_9 2024-03-11T23:09:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_9.status 2024-03-11T23:09:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-03-11T23:09:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:09:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:09:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:09:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:09:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:09:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:09:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 102 = '' 2024-03-11T23:09:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:09:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:09:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_9.status 2024-03-11T23:09:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_9 2024-03-11T23:09:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_9.status 2024-03-11T23:09:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-03-11T23:09:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:05.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:09:05.660 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=102 2024-03-11T23:09:05.660 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 102 = 102 2024-03-11T23:09:05.660 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:09:05.660 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2024-03-11T23:09:05.661 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:05.661 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:05.661 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_9 2024-03-11T23:09:05.661 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:09:05.661 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:09:05.661 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:09:05.661 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:09:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2024-03-11T23:09:06.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_9': 2024-03-11T23:09:06.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:09:06.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:09:06.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:09:06.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 10b3c558b234 2024-03-11T23:09:06.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.10b3c558b234 2024-03-11T23:09:06.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:09:06.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:09:06.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:09:06.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:09:06.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:00:46 2024 2024-03-11T23:09:06.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:00:46 2024 2024-03-11T23:09:06.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:00:46 2024 2024-03-11T23:09:06.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:09:06.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:09:06.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:09:06.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: b02ac2fb-bf75-42d9-bf3f-a8404d4fe797 2024-03-11T23:09:06.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:09:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:09:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_9 snap 2024-03-11T23:09:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_9 2024-03-11T23:09:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:09:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:09:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_9.export 2024-03-11T23:09:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_9.export 2024-03-11T23:09:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_9.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_9.export 2024-03-11T23:09:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_9.export 2024-03-11T23:09:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:09:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_9.export 2024-03-11T23:09:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:09:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_9.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_9.export 2024-03-11T23:09:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_9.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_9.export 2024-03-11T23:09:07.269 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:09:07.270 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:09:07.270 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_10 2024-03-11T23:09:07.270 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_10 snap 2024-03-11T23:09:07.270 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:07.270 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:07.270 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_10 2024-03-11T23:09:07.270 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:09:07.270 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:09:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2024-03-11T23:09:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_10 2024-03-11T23:09:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2024-03-11T23:09:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_10 2024-03-11T23:09:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:09:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:09:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:09:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:09:08.907 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2024-03-11T23:09:08.907 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:08.907 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:08.908 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_10 2024-03-11T23:09:08.908 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:09:08.908 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:09:08.908 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:09:08.908 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2024-03-11T23:09:08.908 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:09:08.908 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:09:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:09:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:09:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:09:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:09:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:09:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:09:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:09:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:09:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:09:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2024-03-11T23:09:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:09:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:09:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:09:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:09:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:09:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:09:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:09:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:09:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:09:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:09:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:09:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:09:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:09:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:09:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:09:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:09:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:09:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:09:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:09:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:09:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:09:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:09:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:09:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:09:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:09:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:09:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:09:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:09:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:09:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:09:08.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:09:08.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:09:08.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:09:08.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2024-03-11T23:09:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-03-11T23:09:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:09:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:09:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:09:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2024-03-11T23:09:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:09:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:09:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:09:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:09:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:09:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:09:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:09:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:09:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:09:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:09:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:09:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:09:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:09:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:09:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:09:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:09:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:09:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:09:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:09:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:09:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:09:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:09:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:09:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:09:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:09:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:09:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:09:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:09:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:09:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:09:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:09:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:09:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:09:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2024-03-11T23:09:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-03-11T23:09:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:09:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:09:09.126 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:09:09.126 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_10 2024-03-11T23:09:09.126 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T23:09:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:09:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:09:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T23:09:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T23:09:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:09:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:09:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:09:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T23:09:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T23:09:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:09:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:09:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T23:09:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:09:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T23:09:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T23:09:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:09:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:09:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T23:09:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T23:09:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:09:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:09:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:09:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:09:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T23:09:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:09:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:09:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T23:09:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T23:09:09.129 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T23:09:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:09:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:09:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:09:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_10 2024-03-11T23:09:09.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:09:09.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:09:09.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:09:09.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:09:09.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:09:09.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:09:09.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:09:09.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:09:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2024-03-11T23:09:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:09:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_10 2024-03-11T23:09:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:09:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:09:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_10 2024-03-11T23:09:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:09:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_10 2024-03-11T23:09:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_10.status 2024-03-11T23:09:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_10.status 2024-03-11T23:09:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:09:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_10.status 2024-03-11T23:09:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_10.status 2024-03-11T23:09:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_10.status 2024-03-11T23:09:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:09:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_10.status 2024-03-11T23:09:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_10 2024-03-11T23:09:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_10 2024-03-11T23:09:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_10 2024-03-11T23:09:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 104 2024-03-11T23:09:09.926 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_10 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_10.status 2024-03-11T23:09:09.926 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:09.926 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:09.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_10 2024-03-11T23:09:09.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_10.status 2024-03-11T23:09:09.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_10 --format xml 2024-03-11T23:09:09.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:09.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:09:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=104 2024-03-11T23:09:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:09:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:09:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:09:10.174 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_10.status 2024-03-11T23:09:10.175 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:10.175 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:10.175 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_10 2024-03-11T23:09:10.175 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_10.status 2024-03-11T23:09:10.175 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-03-11T23:09:10.175 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:10.221 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:09:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=44 2024-03-11T23:09:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 104 = 44 2024-03-11T23:09:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:09:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:09:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_10.status 2024-03-11T23:09:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_10 2024-03-11T23:09:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_10.status 2024-03-11T23:09:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-03-11T23:09:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:09:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=44 2024-03-11T23:09:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 104 = 44 2024-03-11T23:09:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:09:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:09:11.479 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_10.status 2024-03-11T23:09:11.479 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:11.479 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:11.479 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_10 2024-03-11T23:09:11.479 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_10.status 2024-03-11T23:09:11.479 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-03-11T23:09:11.479 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:11.530 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:09:11.530 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:09:11.531 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:09:11.531 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:09:11.531 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:09:11.532 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:09:11.532 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 104 = '' 2024-03-11T23:09:11.532 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:09:11.532 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:09:13.134 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_10.status 2024-03-11T23:09:13.134 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:13.134 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:13.135 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_10 2024-03-11T23:09:13.135 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_10.status 2024-03-11T23:09:13.135 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-03-11T23:09:13.135 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:13.182 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:09:13.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=104 2024-03-11T23:09:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 104 = 104 2024-03-11T23:09:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:09:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2024-03-11T23:09:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_10 2024-03-11T23:09:13.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:09:13.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:09:13.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:09:13.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:09:14.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2024-03-11T23:09:14.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_10': 2024-03-11T23:09:14.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:09:14.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:09:14.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:09:14.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 11ffec52471f 2024-03-11T23:09:14.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.11ffec52471f 2024-03-11T23:09:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:09:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:09:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:09:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:09:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:01:01 2024 2024-03-11T23:09:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:01:01 2024 2024-03-11T23:09:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:01:01 2024 2024-03-11T23:09:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:09:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:09:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:09:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 6e7e7d8e-c0d0-4968-90fa-7731ffa367f1 2024-03-11T23:09:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:09:14.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:09:14.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_10 snap 2024-03-11T23:09:14.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:14.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_10 2024-03-11T23:09:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:09:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:09:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_10.export 2024-03-11T23:09:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_10.export 2024-03-11T23:09:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_10.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_10.export 2024-03-11T23:09:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_10.export 2024-03-11T23:09:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:09:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_10.export 2024-03-11T23:09:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:09:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_10.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_10.export 2024-03-11T23:09:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_10.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_10.export 2024-03-11T23:09:14.739 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:09:14.739 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:09:14.739 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_11 2024-03-11T23:09:14.739 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_11 snap 2024-03-11T23:09:14.739 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:14.739 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:14.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_11 2024-03-11T23:09:14.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:09:14.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:09:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2024-03-11T23:09:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_11 2024-03-11T23:09:14.915 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2024-03-11T23:09:14.915 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:14.915 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:14.915 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_11 2024-03-11T23:09:14.915 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:09:14.915 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:09:14.915 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:09:14.915 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:09:15.915 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2024-03-11T23:09:15.915 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:15.915 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:15.915 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_11 2024-03-11T23:09:15.915 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:09:15.915 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:09:15.915 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:09:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2024-03-11T23:09:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:09:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:09:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:09:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:09:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:09:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:09:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:09:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:09:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:09:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:09:15.919 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:09:15.919 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2024-03-11T23:09:15.919 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:09:15.919 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:09:15.919 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:09:15.919 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:09:15.920 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:09:15.920 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:09:15.920 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:09:15.920 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:09:15.920 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:09:15.920 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:09:15.920 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:09:15.920 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:09:15.920 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:09:15.920 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:09:15.920 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:09:15.920 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:09:15.920 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:09:15.920 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:09:15.921 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:09:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:09:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:09:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:09:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:09:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:09:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:09:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:09:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:09:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:09:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:09:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:09:15.941 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:09:15.941 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:09:15.941 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:09:15.941 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2024-03-11T23:09:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-03-11T23:09:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:09:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:09:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:09:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_11 2024-03-11T23:09:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:09:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:09:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:09:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:09:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:09:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:09:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:09:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:09:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:09:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:09:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:09:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:09:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:09:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:09:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:09:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:09:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:09:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:09:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:09:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:09:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:09:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:09:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:09:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:09:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:09:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:09:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:09:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:09:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:09:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:09:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:09:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:09:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:09:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_11 2024-03-11T23:09:16.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:09:16.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:09:16.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:09:16.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:09:16.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:09:16.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:09:16.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:09:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:09:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2024-03-11T23:09:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:09:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_11 2024-03-11T23:09:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:09:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:09:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_11 2024-03-11T23:09:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:09:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_11 2024-03-11T23:09:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_11.status 2024-03-11T23:09:16.145 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_11.status 2024-03-11T23:09:16.145 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:09:16.146 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_11.status 2024-03-11T23:09:16.146 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_11.status 2024-03-11T23:09:16.146 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_11.status 2024-03-11T23:09:16.146 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:09:16.146 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_11.status 2024-03-11T23:09:16.146 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_11 2024-03-11T23:09:16.146 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:16.146 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:16.146 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_11 2024-03-11T23:09:16.146 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_11 2024-03-11T23:09:16.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 106 2024-03-11T23:09:16.937 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_11 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_11.status 2024-03-11T23:09:16.937 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:16.937 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:16.937 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_11 2024-03-11T23:09:16.937 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_11.status 2024-03-11T23:09:16.937 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_11 --format xml 2024-03-11T23:09:16.937 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:16.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:09:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=106 2024-03-11T23:09:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:09:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:09:16.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:09:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_11.status 2024-03-11T23:09:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_11 2024-03-11T23:09:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_11.status 2024-03-11T23:09:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-03-11T23:09:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:09:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=45 2024-03-11T23:09:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 106 = 45 2024-03-11T23:09:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:09:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:09:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_11.status 2024-03-11T23:09:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_11 2024-03-11T23:09:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_11.status 2024-03-11T23:09:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-03-11T23:09:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:17.679 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:09:17.681 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=45 2024-03-11T23:09:17.681 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 106 = 45 2024-03-11T23:09:17.681 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:09:17.681 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:09:18.482 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_11.status 2024-03-11T23:09:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_11 2024-03-11T23:09:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_11.status 2024-03-11T23:09:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-03-11T23:09:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:18.533 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:09:18.533 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:09:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:09:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:09:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:09:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:09:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 106 = '' 2024-03-11T23:09:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:09:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:09:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_11.status 2024-03-11T23:09:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_11 2024-03-11T23:09:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_11.status 2024-03-11T23:09:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-03-11T23:09:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:09:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=106 2024-03-11T23:09:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 106 = 106 2024-03-11T23:09:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:09:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2024-03-11T23:09:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_11 2024-03-11T23:09:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:09:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:09:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:09:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:09:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2024-03-11T23:09:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_11': 2024-03-11T23:09:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:09:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:09:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:09:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 1260d8dfc0df 2024-03-11T23:09:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.1260d8dfc0df 2024-03-11T23:09:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:09:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:09:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:09:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:09:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:01:08 2024 2024-03-11T23:09:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:01:08 2024 2024-03-11T23:09:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:01:08 2024 2024-03-11T23:09:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:09:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:09:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:09:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 952b2674-c455-44d6-b523-5d182beac4cc 2024-03-11T23:09:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:09:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:09:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_11 snap 2024-03-11T23:09:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_11 2024-03-11T23:09:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:09:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:09:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_11.export 2024-03-11T23:09:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_11.export 2024-03-11T23:09:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_11.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_11.export 2024-03-11T23:09:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_11.export 2024-03-11T23:09:21.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:09:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_11.export 2024-03-11T23:09:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:09:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_11.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_11.export 2024-03-11T23:09:21.728 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_11.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_11.export 2024-03-11T23:09:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:09:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:09:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_12 2024-03-11T23:09:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_12 snap 2024-03-11T23:09:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_12 2024-03-11T23:09:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:09:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:09:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2024-03-11T23:09:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_12 2024-03-11T23:09:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2024-03-11T23:09:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_12 2024-03-11T23:09:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:09:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:09:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:09:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:09:22.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2024-03-11T23:09:22.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:22.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:22.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_12 2024-03-11T23:09:22.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:09:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:09:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:09:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2024-03-11T23:09:22.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:09:22.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:09:22.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:09:22.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:09:22.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:09:22.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:09:22.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:09:22.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:09:22.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:09:22.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:09:22.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:09:22.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2024-03-11T23:09:22.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:09:22.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:09:22.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:09:22.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:09:22.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:09:22.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:09:22.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:09:22.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:09:22.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:09:22.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:09:22.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:09:22.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:09:22.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:09:22.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:09:22.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:09:22.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:09:22.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:09:22.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:09:22.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:09:22.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:09:22.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:09:22.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:09:22.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:09:22.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:09:22.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:09:22.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:09:22.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:09:22.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:09:22.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:09:22.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:09:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:09:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:09:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:09:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2024-03-11T23:09:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-03-11T23:09:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:09:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:09:23.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:09:23.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2024-03-11T23:09:23.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:09:23.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:09:23.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:09:23.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:09:23.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:09:23.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:09:23.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:09:23.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:09:23.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:09:23.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:09:23.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:09:23.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:09:23.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:09:23.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:09:23.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:09:23.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:09:23.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:09:23.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:09:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:09:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:09:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:09:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:09:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:09:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:09:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:09:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:09:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:09:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:09:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:09:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:09:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:09:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:09:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:09:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2024-03-11T23:09:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:09:23.182 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:09:23.182 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:09:23.182 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:09:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:09:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:09:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:09:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:09:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2024-03-11T23:09:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:09:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_12 2024-03-11T23:09:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:09:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:09:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_12 2024-03-11T23:09:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:09:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_12 2024-03-11T23:09:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_12.status 2024-03-11T23:09:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_12.status 2024-03-11T23:09:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:09:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_12.status 2024-03-11T23:09:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_12.status 2024-03-11T23:09:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_12.status 2024-03-11T23:09:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:09:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_12.status 2024-03-11T23:09:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_12 2024-03-11T23:09:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_12 2024-03-11T23:09:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_12 2024-03-11T23:09:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 108 2024-03-11T23:09:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_12 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_12.status 2024-03-11T23:09:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_12 2024-03-11T23:09:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_12.status 2024-03-11T23:09:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_12 --format xml 2024-03-11T23:09:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:09:24.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=108 2024-03-11T23:09:24.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:09:24.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:09:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:09:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_12.status 2024-03-11T23:09:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_12 2024-03-11T23:09:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_12.status 2024-03-11T23:09:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-03-11T23:09:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:09:24.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=46 2024-03-11T23:09:24.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 108 = 46 2024-03-11T23:09:24.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:09:24.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:09:24.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_12.status 2024-03-11T23:09:24.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:24.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_12 2024-03-11T23:09:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_12.status 2024-03-11T23:09:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-03-11T23:09:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:09:24.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=46 2024-03-11T23:09:24.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 108 = 46 2024-03-11T23:09:24.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:09:24.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:09:25.516 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_12.status 2024-03-11T23:09:25.516 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:25.516 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:25.516 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_12 2024-03-11T23:09:25.516 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_12.status 2024-03-11T23:09:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-03-11T23:09:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:25.562 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:09:25.563 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:09:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:09:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:09:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:09:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:09:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 108 = '' 2024-03-11T23:09:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:09:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:09:26.052 INFO:tasks.ceph.cluster2.osd.2.smithi132.stderr:2024-03-11T23:09:26.049+0000 7f6376022640 -1 reset not still connected to 0x56507927a270 2024-03-11T23:09:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_12.status 2024-03-11T23:09:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_12 2024-03-11T23:09:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_12.status 2024-03-11T23:09:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-03-11T23:09:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:09:27.219 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=108 2024-03-11T23:09:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 108 = 108 2024-03-11T23:09:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:09:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2024-03-11T23:09:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_12 2024-03-11T23:09:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:09:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:09:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:09:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:09:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2024-03-11T23:09:28.272 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_12': 2024-03-11T23:09:28.272 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:09:28.272 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:09:28.272 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:09:28.272 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 127ff5a091b3 2024-03-11T23:09:28.273 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.127ff5a091b3 2024-03-11T23:09:28.273 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:09:28.273 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:09:28.273 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:09:28.273 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:09:28.273 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:01:16 2024 2024-03-11T23:09:28.273 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:01:16 2024 2024-03-11T23:09:28.273 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:01:16 2024 2024-03-11T23:09:28.273 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:09:28.273 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:09:28.273 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:09:28.273 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: f99801f5-5b32-4c9d-b6d4-607f161fc716 2024-03-11T23:09:28.273 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:09:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:09:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_12 snap 2024-03-11T23:09:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_12 2024-03-11T23:09:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:09:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:09:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_12.export 2024-03-11T23:09:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_12.export 2024-03-11T23:09:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_12.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_12.export 2024-03-11T23:09:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_12.export 2024-03-11T23:09:28.461 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:09:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_12.export 2024-03-11T23:09:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:09:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_12.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_12.export 2024-03-11T23:09:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_12.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_12.export 2024-03-11T23:09:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:09:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:09:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_13 2024-03-11T23:09:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_13 snap 2024-03-11T23:09:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_13 2024-03-11T23:09:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:09:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:09:29.962 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2024-03-11T23:09:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_13 2024-03-11T23:09:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2024-03-11T23:09:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_13 2024-03-11T23:09:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:09:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:09:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:09:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:09:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2024-03-11T23:09:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_13 2024-03-11T23:09:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:09:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:09:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:09:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2024-03-11T23:09:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:09:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:09:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:09:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:09:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:09:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:09:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:09:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:09:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:09:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:09:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:09:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2024-03-11T23:09:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:09:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:09:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:09:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:09:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:09:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:09:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:09:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:09:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:09:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:09:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:09:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:09:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:09:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:09:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:09:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:09:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:09:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:09:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:09:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:09:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:09:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:09:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:09:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:09:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:09:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:09:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:09:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:09:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:09:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:09:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:09:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:09:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:09:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2024-03-11T23:09:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:09:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:09:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:09:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:09:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:09:31.092 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:09:31.092 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:09:31.092 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:09:31.092 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2024-03-11T23:09:31.092 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:09:31.092 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:31.092 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:31.092 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_13 2024-03-11T23:09:31.092 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:09:31.092 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:09:31.092 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_13 2024-03-11T23:09:31.092 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:09:31.092 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:31.093 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:31.093 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_13 2024-03-11T23:09:31.093 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_13.status 2024-03-11T23:09:31.093 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_13.status 2024-03-11T23:09:31.093 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:09:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_13.status 2024-03-11T23:09:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_13.status 2024-03-11T23:09:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_13.status 2024-03-11T23:09:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:09:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_13.status 2024-03-11T23:09:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_13 2024-03-11T23:09:31.095 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:31.095 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:31.095 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_13 2024-03-11T23:09:31.095 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_13 2024-03-11T23:09:31.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 110 2024-03-11T23:09:31.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_13 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_13.status 2024-03-11T23:09:31.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:31.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:31.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_13 2024-03-11T23:09:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_13.status 2024-03-11T23:09:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_13 --format xml 2024-03-11T23:09:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:32.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:09:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=110 2024-03-11T23:09:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:09:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:09:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:09:32.228 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_13.status 2024-03-11T23:09:32.228 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_13 2024-03-11T23:09:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_13.status 2024-03-11T23:09:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-03-11T23:09:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:32.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:09:32.281 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=47 2024-03-11T23:09:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 110 = 47 2024-03-11T23:09:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:09:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:09:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_13.status 2024-03-11T23:09:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_13 2024-03-11T23:09:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_13.status 2024-03-11T23:09:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-03-11T23:09:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:09:32.736 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=47 2024-03-11T23:09:32.736 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 110 = 47 2024-03-11T23:09:32.736 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:09:32.736 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:09:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_13.status 2024-03-11T23:09:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_13 2024-03-11T23:09:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_13.status 2024-03-11T23:09:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-03-11T23:09:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:09:33.591 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:09:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:09:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:09:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:09:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:09:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 110 = '' 2024-03-11T23:09:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:09:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:09:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_13.status 2024-03-11T23:09:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:35.195 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_13 2024-03-11T23:09:35.195 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_13.status 2024-03-11T23:09:35.195 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-03-11T23:09:35.195 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:09:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=110 2024-03-11T23:09:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 110 = 110 2024-03-11T23:09:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:09:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2024-03-11T23:09:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_13 2024-03-11T23:09:35.250 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:09:35.250 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:09:35.250 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:09:35.250 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:09:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2024-03-11T23:09:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_13': 2024-03-11T23:09:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:09:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:09:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:09:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 10b3ca55c6a1 2024-03-11T23:09:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.10b3ca55c6a1 2024-03-11T23:09:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:09:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:09:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:09:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:09:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:01:29 2024 2024-03-11T23:09:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:01:29 2024 2024-03-11T23:09:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:01:29 2024 2024-03-11T23:09:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:09:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:09:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:09:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 4e5cb2c2-5e2b-44b7-95cc-f0928b52a2e0 2024-03-11T23:09:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:09:36.299 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:09:36.299 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_13 snap 2024-03-11T23:09:36.299 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:36.299 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_13 2024-03-11T23:09:36.299 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:09:36.299 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:09:36.299 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_13.export 2024-03-11T23:09:36.299 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_13.export 2024-03-11T23:09:36.299 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_13.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_13.export 2024-03-11T23:09:36.299 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_13.export 2024-03-11T23:09:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:09:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_13.export 2024-03-11T23:09:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:09:36.694 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_13.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_13.export 2024-03-11T23:09:36.747 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_13.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_13.export 2024-03-11T23:09:36.778 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:09:36.778 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:09:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_14 2024-03-11T23:09:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_14 snap 2024-03-11T23:09:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_14 2024-03-11T23:09:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:09:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:09:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2024-03-11T23:09:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_14 2024-03-11T23:09:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2024-03-11T23:09:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:36.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_14 2024-03-11T23:09:36.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:09:36.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:09:36.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:09:36.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:09:37.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2024-03-11T23:09:37.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:37.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:37.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_14 2024-03-11T23:09:37.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:09:37.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:09:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:09:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2024-03-11T23:09:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:09:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:09:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:09:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:09:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:09:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:09:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:09:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:09:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:09:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:09:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:09:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2024-03-11T23:09:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:09:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:09:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:09:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:09:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:09:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:09:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:09:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:09:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:09:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:09:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:09:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:09:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:09:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:09:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:09:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:09:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:09:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:09:37.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:09:37.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:09:37.975 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:09:37.975 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:09:37.975 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:09:37.975 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:09:37.975 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:09:37.975 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:09:37.975 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:09:37.975 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:09:37.975 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:09:37.975 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:09:37.994 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:09:37.994 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:09:37.994 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:09:37.994 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2024-03-11T23:09:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:09:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:09:38.093 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:09:38.093 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:09:38.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:09:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:09:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:09:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:09:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2024-03-11T23:09:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:09:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_14 2024-03-11T23:09:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:09:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:09:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_14 2024-03-11T23:09:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:09:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_14 2024-03-11T23:09:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_14.status 2024-03-11T23:09:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_14.status 2024-03-11T23:09:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:09:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_14.status 2024-03-11T23:09:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_14.status 2024-03-11T23:09:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_14.status 2024-03-11T23:09:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:09:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_14.status 2024-03-11T23:09:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_14 2024-03-11T23:09:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_14 2024-03-11T23:09:38.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_14 2024-03-11T23:09:38.913 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-03-11T23:09:38.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-03-11T23:09:38.914 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-03-11T23:09:38.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr:2024-03-11T23:09:38.911+0000 7f9c6779d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 51586) UID: 1000 2024-03-11T23:09:38.986 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 112 2024-03-11T23:09:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_14 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_14.status 2024-03-11T23:09:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_14 2024-03-11T23:09:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_14.status 2024-03-11T23:09:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_14 --format xml 2024-03-11T23:09:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:09:39.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=112 2024-03-11T23:09:39.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:09:39.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:09:39.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:09:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_14.status 2024-03-11T23:09:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_14 2024-03-11T23:09:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_14.status 2024-03-11T23:09:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-03-11T23:09:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:09:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=48 2024-03-11T23:09:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 112 = 48 2024-03-11T23:09:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:09:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:09:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_14.status 2024-03-11T23:09:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_14 2024-03-11T23:09:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_14.status 2024-03-11T23:09:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-03-11T23:09:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:39.944 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:09:39.944 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:09:39.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:09:39.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:09:39.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:09:39.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:09:39.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 112 = '' 2024-03-11T23:09:39.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:09:39.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:09:40.748 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_14.status 2024-03-11T23:09:40.748 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:40.748 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:40.748 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_14 2024-03-11T23:09:40.748 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_14.status 2024-03-11T23:09:40.748 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-03-11T23:09:40.748 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:09:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:09:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:09:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:09:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:09:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:09:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 112 = '' 2024-03-11T23:09:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:09:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:09:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_14.status 2024-03-11T23:09:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_14 2024-03-11T23:09:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_14.status 2024-03-11T23:09:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-03-11T23:09:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:42.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:09:42.453 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=112 2024-03-11T23:09:42.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 112 = 112 2024-03-11T23:09:42.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:09:42.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2024-03-11T23:09:42.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:42.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:42.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_14 2024-03-11T23:09:42.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:09:42.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:09:42.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:09:42.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:09:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2024-03-11T23:09:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_14': 2024-03-11T23:09:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:09:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:09:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:09:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 11fffc2fd94b 2024-03-11T23:09:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.11fffc2fd94b 2024-03-11T23:09:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:09:43.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:09:43.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:09:43.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:09:43.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:01:41 2024 2024-03-11T23:09:43.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:01:41 2024 2024-03-11T23:09:43.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:01:41 2024 2024-03-11T23:09:43.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:09:43.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:09:43.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:09:43.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: e8312f3c-c1b7-4832-92dc-03abff4ac406 2024-03-11T23:09:43.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:09:43.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:09:43.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_14 snap 2024-03-11T23:09:43.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:43.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_14 2024-03-11T23:09:43.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:09:43.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:09:43.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_14.export 2024-03-11T23:09:43.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_14.export 2024-03-11T23:09:43.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_14.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_14.export 2024-03-11T23:09:43.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_14.export 2024-03-11T23:09:43.681 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:09:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_14.export 2024-03-11T23:09:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:09:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_14.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_14.export 2024-03-11T23:09:43.987 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_14.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_14.export 2024-03-11T23:09:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:09:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:09:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_15 2024-03-11T23:09:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_15 snap 2024-03-11T23:09:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:44.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:44.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_15 2024-03-11T23:09:44.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:09:44.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:09:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2024-03-11T23:09:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_15 2024-03-11T23:09:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2024-03-11T23:09:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:44.985 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_15 2024-03-11T23:09:44.985 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:09:44.985 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:09:44.985 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:09:44.985 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:09:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2024-03-11T23:09:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_15 2024-03-11T23:09:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:09:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:09:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:09:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2024-03-11T23:09:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:09:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:09:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:09:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:09:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:09:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:09:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:09:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:09:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:09:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:09:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:09:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2024-03-11T23:09:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:09:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:09:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:09:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:09:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:09:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:09:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:09:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:09:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:09:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:09:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:09:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:09:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:09:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:09:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:09:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:09:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:09:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:09:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:09:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:09:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:09:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:09:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:09:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:09:45.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:09:45.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:09:45.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:09:45.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:09:45.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:09:45.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:09:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:09:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:09:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:09:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2024-03-11T23:09:46.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-03-11T23:09:46.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:09:46.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:09:46.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:09:46.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_15 2024-03-11T23:09:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:09:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:09:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:09:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:09:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:09:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:09:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:09:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:09:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:09:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:09:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:09:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:09:46.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:09:46.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:09:46.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:09:46.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:09:46.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:09:46.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:09:46.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:09:46.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:09:46.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:09:46.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:09:46.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:09:46.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:09:46.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:09:46.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:09:46.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:09:46.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:09:46.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:09:46.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:09:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:09:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:09:46.123 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:09:46.123 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_15 2024-03-11T23:09:46.215 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:09:46.215 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:09:46.215 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:09:46.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:09:46.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:09:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:09:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:09:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:09:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2024-03-11T23:09:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:09:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_15 2024-03-11T23:09:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:09:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:09:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_15 2024-03-11T23:09:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:09:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_15 2024-03-11T23:09:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_15.status 2024-03-11T23:09:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_15.status 2024-03-11T23:09:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:09:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_15.status 2024-03-11T23:09:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_15.status 2024-03-11T23:09:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_15.status 2024-03-11T23:09:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:09:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_15.status 2024-03-11T23:09:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_15 2024-03-11T23:09:46.221 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:46.221 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:46.221 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_15 2024-03-11T23:09:46.221 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_15 2024-03-11T23:09:46.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 114 2024-03-11T23:09:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_15 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_15.status 2024-03-11T23:09:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_15 2024-03-11T23:09:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_15.status 2024-03-11T23:09:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_15 --format xml 2024-03-11T23:09:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:09:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=114 2024-03-11T23:09:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:09:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:09:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:09:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_15.status 2024-03-11T23:09:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_15 2024-03-11T23:09:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_15.status 2024-03-11T23:09:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-03-11T23:09:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:46.609 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:09:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=49 2024-03-11T23:09:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 114 = 49 2024-03-11T23:09:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:09:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:09:46.915 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-03-11T23:09:46.915 INFO:teuthology.orchestra.run:waiting for 600 2024-03-11T23:09:46.916 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-03-11T23:09:46.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-03-11T23:09:46.916 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-03-11T23:09:46.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-03-11T23:09:46.919 INFO:tasks.rbd_mirror_thrash:waiting for 81 secs before thrashing 2024-03-11T23:09:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_15.status 2024-03-11T23:09:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_15 2024-03-11T23:09:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_15.status 2024-03-11T23:09:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-03-11T23:09:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:47.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:09:47.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:09:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:09:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:09:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:09:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:09:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 114 = '' 2024-03-11T23:09:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:09:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:09:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_15.status 2024-03-11T23:09:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_15 2024-03-11T23:09:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_15.status 2024-03-11T23:09:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-03-11T23:09:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:09:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:09:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:09:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:09:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:09:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:09:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 114 = '' 2024-03-11T23:09:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:09:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:09:49.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_15.status 2024-03-11T23:09:49.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:49.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:49.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_15 2024-03-11T23:09:49.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_15.status 2024-03-11T23:09:49.550 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-03-11T23:09:49.550 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:09:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=114 2024-03-11T23:09:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 114 = 114 2024-03-11T23:09:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:09:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2024-03-11T23:09:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_15 2024-03-11T23:09:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:09:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:09:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:09:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:09:50.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2024-03-11T23:09:50.649 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_15': 2024-03-11T23:09:50.649 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:09:50.649 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:09:50.649 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:09:50.649 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 128861d648b 2024-03-11T23:09:50.649 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.128861d648b 2024-03-11T23:09:50.649 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:09:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:09:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:09:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:09:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:01:57 2024 2024-03-11T23:09:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:01:57 2024 2024-03-11T23:09:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:01:57 2024 2024-03-11T23:09:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:09:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:09:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:09:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: bfe4c714-e6df-48ee-b9ba-69d0b7a5f35b 2024-03-11T23:09:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:09:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:09:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_15 snap 2024-03-11T23:09:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_15 2024-03-11T23:09:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:09:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:09:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_15.export 2024-03-11T23:09:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_15.export 2024-03-11T23:09:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_15.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_15.export 2024-03-11T23:09:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_15.export 2024-03-11T23:09:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:09:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_15.export 2024-03-11T23:09:51.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:09:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_15.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_15.export 2024-03-11T23:09:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_15.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_15.export 2024-03-11T23:09:51.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:09:51.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:09:51.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_16 2024-03-11T23:09:51.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_16 snap 2024-03-11T23:09:51.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:51.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:51.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_16 2024-03-11T23:09:51.167 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:09:51.167 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:09:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2024-03-11T23:09:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_16 2024-03-11T23:09:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2024-03-11T23:09:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_16 2024-03-11T23:09:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:09:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:09:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:09:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:09:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2024-03-11T23:09:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_16 2024-03-11T23:09:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:09:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:09:52.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:09:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2024-03-11T23:09:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:09:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:09:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:09:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:09:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:09:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:09:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:09:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:09:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:09:52.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:09:52.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:09:52.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2024-03-11T23:09:52.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:09:52.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:09:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:09:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:09:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:09:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:09:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:09:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:09:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:09:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:09:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:09:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:09:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:09:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:09:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:09:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:09:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:09:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:09:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:09:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:09:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:09:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:09:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:09:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:09:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:09:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:09:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:09:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:09:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:09:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:09:52.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:09:52.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:09:52.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:09:52.338 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2024-03-11T23:09:52.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:09:52.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:09:52.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:09:52.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:09:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:09:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:09:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:09:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:09:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2024-03-11T23:09:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:09:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_16 2024-03-11T23:09:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:09:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:09:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_16 2024-03-11T23:09:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:09:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_16 2024-03-11T23:09:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_16.status 2024-03-11T23:09:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_16.status 2024-03-11T23:09:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:09:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_16.status 2024-03-11T23:09:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_16.status 2024-03-11T23:09:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_16.status 2024-03-11T23:09:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:09:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_16.status 2024-03-11T23:09:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_16 2024-03-11T23:09:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:52.440 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:52.440 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_16 2024-03-11T23:09:52.440 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_16 2024-03-11T23:09:53.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 116 2024-03-11T23:09:53.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_16 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_16.status 2024-03-11T23:09:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_16 2024-03-11T23:09:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_16.status 2024-03-11T23:09:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_16 --format xml 2024-03-11T23:09:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:53.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:09:53.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=116 2024-03-11T23:09:53.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:09:53.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:09:53.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:09:53.783 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_16.status 2024-03-11T23:09:53.783 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_16 2024-03-11T23:09:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_16.status 2024-03-11T23:09:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-03-11T23:09:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:53.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:09:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=50 2024-03-11T23:09:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 116 = 50 2024-03-11T23:09:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:09:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:09:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_16.status 2024-03-11T23:09:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_16 2024-03-11T23:09:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_16.status 2024-03-11T23:09:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-03-11T23:09:54.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:54.288 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:09:54.288 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:09:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:09:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:09:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:09:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:09:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 116 = '' 2024-03-11T23:09:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:09:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:09:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_16.status 2024-03-11T23:09:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_16 2024-03-11T23:09:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_16.status 2024-03-11T23:09:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:55.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-03-11T23:09:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:09:55.178 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:09:55.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:09:55.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:09:55.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:09:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:09:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 116 = '' 2024-03-11T23:09:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:09:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:09:56.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_16.status 2024-03-11T23:09:56.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:56.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:56.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_16 2024-03-11T23:09:56.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_16.status 2024-03-11T23:09:56.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-03-11T23:09:56.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:09:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:09:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=116 2024-03-11T23:09:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 116 = 116 2024-03-11T23:09:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:09:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2024-03-11T23:09:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:56.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_16 2024-03-11T23:09:56.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:09:56.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:09:56.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:09:56.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:09:57.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2024-03-11T23:09:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_16': 2024-03-11T23:09:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:09:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:09:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:09:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 1288929905ac 2024-03-11T23:09:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.1288929905ac 2024-03-11T23:09:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:09:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:09:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:09:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:09:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:02:06 2024 2024-03-11T23:09:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:02:06 2024 2024-03-11T23:09:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:02:06 2024 2024-03-11T23:09:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:09:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:09:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:09:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 8afe84ee-3bfc-43a7-b04d-7925c8799653 2024-03-11T23:09:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:09:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:09:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_16 snap 2024-03-11T23:09:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_16 2024-03-11T23:09:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:09:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:09:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_16.export 2024-03-11T23:09:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_16.export 2024-03-11T23:09:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_16.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_16.export 2024-03-11T23:09:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_16.export 2024-03-11T23:09:58.084 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:09:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_16.export 2024-03-11T23:09:58.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:09:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_16.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_16.export 2024-03-11T23:09:58.398 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_16.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_16.export 2024-03-11T23:09:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:09:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:09:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_17 2024-03-11T23:09:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_17 snap 2024-03-11T23:09:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:09:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_17 2024-03-11T23:09:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:09:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:09:59.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2024-03-11T23:09:59.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:59.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:59.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_17 2024-03-11T23:09:59.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2024-03-11T23:09:59.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:09:59.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:09:59.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_17 2024-03-11T23:09:59.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:09:59.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:09:59.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:09:59.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:10:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2024-03-11T23:10:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_17 2024-03-11T23:10:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:10:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:10:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:10:00.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2024-03-11T23:10:00.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:10:00.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:10:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:10:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:10:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:10:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:10:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:10:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:10:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:10:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:10:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2024-03-11T23:10:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:10:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:10:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:10:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:10:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:10:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:10:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:10:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:10:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:10:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:10:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:10:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:10:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:10:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:10:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:10:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:10:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:10:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:10:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:10:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:10:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:10:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:10:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:10:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:10:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:10:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:10:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:10:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:10:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:10:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:10:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2024-03-11T23:10:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:10:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:10:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:10:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:10:00.466 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:10:00.467 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:10:00.468 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:10:00.468 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:10:00.468 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2024-03-11T23:10:00.468 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:10:00.468 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:10:00.468 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:00.468 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_17 2024-03-11T23:10:00.468 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:10:00.468 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:10:00.468 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_17 2024-03-11T23:10:00.468 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:10:00.468 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:10:00.468 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:00.468 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_17 2024-03-11T23:10:00.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_17.status 2024-03-11T23:10:00.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_17.status 2024-03-11T23:10:00.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:10:00.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_17.status 2024-03-11T23:10:00.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_17.status 2024-03-11T23:10:00.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_17.status 2024-03-11T23:10:00.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:10:00.470 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_17.status 2024-03-11T23:10:00.470 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_17 2024-03-11T23:10:00.470 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:10:00.470 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:00.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_17 2024-03-11T23:10:00.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_17 2024-03-11T23:10:01.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 118 2024-03-11T23:10:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_17 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_17.status 2024-03-11T23:10:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:10:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_17 2024-03-11T23:10:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_17.status 2024-03-11T23:10:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_17 --format xml 2024-03-11T23:10:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:10:01.409 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:10:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=118 2024-03-11T23:10:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:10:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:10:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:10:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_17.status 2024-03-11T23:10:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_17 2024-03-11T23:10:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_17.status 2024-03-11T23:10:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-03-11T23:10:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:10:01.662 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:10:01.663 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=51 2024-03-11T23:10:01.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 118 = 51 2024-03-11T23:10:01.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:10:01.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:10:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_17.status 2024-03-11T23:10:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_17 2024-03-11T23:10:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_17.status 2024-03-11T23:10:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-03-11T23:10:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:10:02.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:10:02.116 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:10:02.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:10:02.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:10:02.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:10:02.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:10:02.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 118 = '' 2024-03-11T23:10:02.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:10:02.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:10:02.919 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_17.status 2024-03-11T23:10:02.919 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:02.920 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:02.920 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_17 2024-03-11T23:10:02.920 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_17.status 2024-03-11T23:10:02.920 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-03-11T23:10:02.920 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:10:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:10:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:10:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:10:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:10:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:10:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:10:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 118 = '' 2024-03-11T23:10:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:10:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:10:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_17.status 2024-03-11T23:10:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_17 2024-03-11T23:10:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_17.status 2024-03-11T23:10:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-03-11T23:10:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:10:04.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:10:04.627 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=118 2024-03-11T23:10:04.627 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 118 = 118 2024-03-11T23:10:04.628 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:10:04.628 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2024-03-11T23:10:04.628 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:04.628 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:04.628 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_17 2024-03-11T23:10:04.628 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:10:04.628 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:10:04.628 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:10:04.628 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:10:05.628 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2024-03-11T23:10:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_17': 2024-03-11T23:10:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:10:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:10:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:10:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 128846e6a5ad 2024-03-11T23:10:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.128846e6a5ad 2024-03-11T23:10:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:10:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:10:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:10:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:10:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:02:13 2024 2024-03-11T23:10:05.674 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:02:13 2024 2024-03-11T23:10:05.674 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:02:13 2024 2024-03-11T23:10:05.674 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:10:05.674 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:10:05.674 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:10:05.674 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 2a3f0f67-66fa-49bb-9d97-83ed1ad32bc2 2024-03-11T23:10:05.674 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:10:05.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:10:05.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_17 snap 2024-03-11T23:10:05.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:05.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_17 2024-03-11T23:10:05.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:10:05.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:10:05.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_17.export 2024-03-11T23:10:05.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_17.export 2024-03-11T23:10:05.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_17.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_17.export 2024-03-11T23:10:05.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_17.export 2024-03-11T23:10:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:10:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_17.export 2024-03-11T23:10:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:10:06.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_17.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_17.export 2024-03-11T23:10:06.153 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_17.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_17.export 2024-03-11T23:10:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:10:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:10:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_18 2024-03-11T23:10:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_18 snap 2024-03-11T23:10:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:10:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_18 2024-03-11T23:10:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:10:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:10:06.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2024-03-11T23:10:06.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:06.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:06.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_18 2024-03-11T23:10:06.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2024-03-11T23:10:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_18 2024-03-11T23:10:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:10:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:10:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:10:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:10:07.359 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2024-03-11T23:10:07.359 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:07.359 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:07.359 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_18 2024-03-11T23:10:07.359 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:10:07.359 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:10:07.359 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:10:07.359 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2024-03-11T23:10:07.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:10:07.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:07.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:10:07.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:10:07.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:10:07.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:10:07.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:10:07.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:10:07.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:10:07.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:10:07.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:10:07.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2024-03-11T23:10:07.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:10:07.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:07.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:10:07.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:10:07.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:10:07.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:10:07.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:10:07.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:10:07.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:10:07.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:10:07.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:10:07.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:07.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:10:07.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:10:07.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:10:07.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:10:07.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:10:07.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:10:07.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:10:07.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:10:07.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:10:07.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:10:07.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:07.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:10:07.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:10:07.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:10:07.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:10:07.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:10:07.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:10:07.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:10:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:10:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:10:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:10:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2024-03-11T23:10:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:10:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:10:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:10:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:10:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:10:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:10:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:10:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:10:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2024-03-11T23:10:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:10:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:10:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_18 2024-03-11T23:10:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:10:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:10:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_18 2024-03-11T23:10:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:10:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:10:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_18 2024-03-11T23:10:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_18.status 2024-03-11T23:10:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:10:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_18.status 2024-03-11T23:10:07.487 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_18.status 2024-03-11T23:10:07.487 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_18.status 2024-03-11T23:10:07.487 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_18.status 2024-03-11T23:10:07.487 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:10:07.487 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_18.status 2024-03-11T23:10:07.487 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_18 2024-03-11T23:10:07.487 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:10:07.487 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:07.487 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_18 2024-03-11T23:10:07.487 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_18 2024-03-11T23:10:08.403 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 120 2024-03-11T23:10:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_18 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_18.status 2024-03-11T23:10:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:10:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_18 2024-03-11T23:10:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_18.status 2024-03-11T23:10:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_18 --format xml 2024-03-11T23:10:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:10:08.456 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:10:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=120 2024-03-11T23:10:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:10:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:10:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:10:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_18.status 2024-03-11T23:10:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_18 2024-03-11T23:10:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_18.status 2024-03-11T23:10:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-03-11T23:10:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:10:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:10:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=52 2024-03-11T23:10:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 120 = 52 2024-03-11T23:10:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:10:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:10:09.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_18.status 2024-03-11T23:10:09.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:09.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:09.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_18 2024-03-11T23:10:09.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_18.status 2024-03-11T23:10:09.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-03-11T23:10:09.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:10:09.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:10:09.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:10:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:10:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:10:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:10:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:10:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 120 = '' 2024-03-11T23:10:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:10:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:10:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_18.status 2024-03-11T23:10:09.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:09.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:09.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_18 2024-03-11T23:10:09.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_18.status 2024-03-11T23:10:09.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-03-11T23:10:09.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:10:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:10:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:10:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:10:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:10:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:10:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:10:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 120 = '' 2024-03-11T23:10:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:10:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:10:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_18.status 2024-03-11T23:10:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_18 2024-03-11T23:10:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_18.status 2024-03-11T23:10:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-03-11T23:10:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:10:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:10:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=120 2024-03-11T23:10:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 120 = 120 2024-03-11T23:10:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:10:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2024-03-11T23:10:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_18 2024-03-11T23:10:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:10:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:10:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:10:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:10:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2024-03-11T23:10:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_18': 2024-03-11T23:10:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:10:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:10:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:10:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 11ffaa4f59a 2024-03-11T23:10:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.11ffaa4f59a 2024-03-11T23:10:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:10:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:10:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:10:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:10:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:02:26 2024 2024-03-11T23:10:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:02:26 2024 2024-03-11T23:10:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:02:26 2024 2024-03-11T23:10:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:10:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:10:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:10:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 387adbd8-be3f-401f-ba8e-7062c5a54dfd 2024-03-11T23:10:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:10:12.760 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:10:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_18 snap 2024-03-11T23:10:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_18 2024-03-11T23:10:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:10:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:10:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_18.export 2024-03-11T23:10:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_18.export 2024-03-11T23:10:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_18.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_18.export 2024-03-11T23:10:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_18.export 2024-03-11T23:10:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:10:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_18.export 2024-03-11T23:10:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:10:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_18.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_18.export 2024-03-11T23:10:13.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_18.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_18.export 2024-03-11T23:10:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:10:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:10:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_19 2024-03-11T23:10:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_19 snap 2024-03-11T23:10:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:10:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_19 2024-03-11T23:10:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:10:13.296 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:10:13.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2024-03-11T23:10:13.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:13.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:13.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_19 2024-03-11T23:10:13.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2024-03-11T23:10:13.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:13.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:13.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_19 2024-03-11T23:10:13.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:10:13.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:10:13.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:10:13.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:10:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2024-03-11T23:10:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_19 2024-03-11T23:10:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:10:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:10:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:10:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2024-03-11T23:10:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:10:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:10:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:10:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:10:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:10:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:10:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:10:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:10:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:10:14.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:10:14.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2024-03-11T23:10:14.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:10:14.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:10:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:10:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:10:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:10:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:10:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:10:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:10:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:10:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:10:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:10:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:10:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:10:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:10:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:10:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:10:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:10:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:10:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:10:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:10:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:10:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:10:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:10:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:10:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:10:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:10:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:10:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:10:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:10:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:10:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2024-03-11T23:10:14.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2024-03-11T23:10:14.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:10:14.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:10:14.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:10:14.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_19 2024-03-11T23:10:14.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:10:14.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:14.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:10:14.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:10:14.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:10:14.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:10:14.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:10:14.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:10:14.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:10:14.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:10:14.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:10:14.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:14.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:10:14.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:10:14.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:10:14.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:10:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:10:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:10:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:10:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:10:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:10:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:10:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:10:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:10:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:10:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:10:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:10:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:10:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:10:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:10:14.492 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:10:14.492 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:10:14.492 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_19 2024-03-11T23:10:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2024-03-11T23:10:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:10:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:10:14.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:10:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_19 2024-03-11T23:10:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T23:10:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:10:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T23:10:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T23:10:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:10:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:10:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:10:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T23:10:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T23:10:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:10:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T23:10:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:10:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T23:10:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T23:10:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:10:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:10:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T23:10:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T23:10:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:10:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:10:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:10:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T23:10:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:10:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:10:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T23:10:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T23:10:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T23:10:14.597 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:10:14.597 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:10:14.597 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:10:14.597 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_19 2024-03-11T23:10:14.686 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:10:14.687 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:10:14.687 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:10:14.687 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:10:14.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:10:14.689 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:10:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:10:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:10:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2024-03-11T23:10:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:10:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:10:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_19 2024-03-11T23:10:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:10:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:10:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_19 2024-03-11T23:10:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:10:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:10:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_19 2024-03-11T23:10:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_19.status 2024-03-11T23:10:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_19.status 2024-03-11T23:10:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:10:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_19.status 2024-03-11T23:10:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_19.status 2024-03-11T23:10:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_19.status 2024-03-11T23:10:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:10:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_19.status 2024-03-11T23:10:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_19 2024-03-11T23:10:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:10:14.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:14.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_19 2024-03-11T23:10:14.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_19 2024-03-11T23:10:15.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 122 2024-03-11T23:10:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_19 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_19.status 2024-03-11T23:10:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:10:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:15.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_19 2024-03-11T23:10:15.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_19.status 2024-03-11T23:10:15.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_19 --format xml 2024-03-11T23:10:15.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:10:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:10:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=122 2024-03-11T23:10:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:10:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:10:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:10:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_19.status 2024-03-11T23:10:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_19 2024-03-11T23:10:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_19.status 2024-03-11T23:10:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-03-11T23:10:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:10:15.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:10:15.673 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=53 2024-03-11T23:10:15.673 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 122 = 53 2024-03-11T23:10:15.673 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:10:15.673 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:10:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_19.status 2024-03-11T23:10:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_19 2024-03-11T23:10:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_19.status 2024-03-11T23:10:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-03-11T23:10:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:10:16.120 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:10:16.120 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:10:16.121 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:10:16.121 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:10:16.121 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:10:16.121 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:10:16.121 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 122 = '' 2024-03-11T23:10:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:10:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:10:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_19.status 2024-03-11T23:10:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_19 2024-03-11T23:10:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_19.status 2024-03-11T23:10:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-03-11T23:10:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:10:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:10:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:10:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:10:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:10:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:10:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:10:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 122 = '' 2024-03-11T23:10:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:10:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:10:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_19.status 2024-03-11T23:10:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_19 2024-03-11T23:10:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_19.status 2024-03-11T23:10:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-03-11T23:10:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:10:18.628 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:10:18.630 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=122 2024-03-11T23:10:18.630 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 122 = 122 2024-03-11T23:10:18.630 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:10:18.630 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2024-03-11T23:10:18.630 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:18.630 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:18.631 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_19 2024-03-11T23:10:18.631 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:10:18.631 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:10:18.631 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:10:18.631 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:10:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2024-03-11T23:10:19.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_19': 2024-03-11T23:10:19.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:10:19.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:10:19.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:10:19.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 1260a84b1db 2024-03-11T23:10:19.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.1260a84b1db 2024-03-11T23:10:19.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:10:19.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:10:19.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:10:19.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:10:19.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:02:33 2024 2024-03-11T23:10:19.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:02:33 2024 2024-03-11T23:10:19.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:02:33 2024 2024-03-11T23:10:19.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:10:19.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:10:19.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:10:19.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 06685589-ab88-4e65-be67-9272ee49920c 2024-03-11T23:10:19.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:10:19.680 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:10:19.681 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_19 snap 2024-03-11T23:10:19.681 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:19.681 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_19 2024-03-11T23:10:19.681 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:10:19.681 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:10:19.681 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_19.export 2024-03-11T23:10:19.682 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_19.export 2024-03-11T23:10:19.682 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_19.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_19.export 2024-03-11T23:10:19.682 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_19.export 2024-03-11T23:10:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:10:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_19.export 2024-03-11T23:10:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:10:20.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_19.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_19.export 2024-03-11T23:10:20.172 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_19.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_19.export 2024-03-11T23:10:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:10:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:10:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_20 2024-03-11T23:10:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_20 snap 2024-03-11T23:10:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:10:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_20 2024-03-11T23:10:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:10:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:10:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2024-03-11T23:10:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_20 2024-03-11T23:10:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2024-03-11T23:10:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_20 2024-03-11T23:10:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:10:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:10:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:10:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:10:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2024-03-11T23:10:21.919 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:21.919 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:21.919 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_20 2024-03-11T23:10:21.919 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:10:21.919 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:10:21.919 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:10:21.920 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2024-03-11T23:10:21.920 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:10:21.920 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:21.920 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:10:21.920 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:10:21.920 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:10:21.920 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:10:21.920 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:10:21.920 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:10:21.921 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:10:21.921 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:10:21.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:10:21.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2024-03-11T23:10:21.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:10:21.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:21.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:10:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:10:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:10:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:10:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:10:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:10:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:10:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:10:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:10:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:10:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:10:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:10:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:10:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:10:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:10:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:10:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:10:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:10:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:10:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:21.926 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:10:21.926 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:10:21.926 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:10:21.926 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:10:21.926 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:10:21.926 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:10:21.926 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:10:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:10:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:10:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:10:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2024-03-11T23:10:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:10:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:10:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:10:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:10:22.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:10:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:10:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:10:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:10:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2024-03-11T23:10:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:10:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:10:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_20 2024-03-11T23:10:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:10:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:10:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_20 2024-03-11T23:10:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:10:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:10:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_20 2024-03-11T23:10:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_20.status 2024-03-11T23:10:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_20.status 2024-03-11T23:10:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:10:22.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_20.status 2024-03-11T23:10:22.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_20.status 2024-03-11T23:10:22.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_20.status 2024-03-11T23:10:22.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:10:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_20.status 2024-03-11T23:10:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_20 2024-03-11T23:10:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:10:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_20 2024-03-11T23:10:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_20 2024-03-11T23:10:22.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 124 2024-03-11T23:10:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_20 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_20.status 2024-03-11T23:10:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:10:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_20 2024-03-11T23:10:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_20.status 2024-03-11T23:10:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_20 --format xml 2024-03-11T23:10:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:10:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:10:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=124 2024-03-11T23:10:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:10:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:10:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:10:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_20.status 2024-03-11T23:10:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_20 2024-03-11T23:10:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_20.status 2024-03-11T23:10:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-03-11T23:10:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:10:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:10:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:10:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:10:23.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:10:23.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:10:23.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:10:23.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 124 = '' 2024-03-11T23:10:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:10:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:10:23.632 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_20.status 2024-03-11T23:10:23.633 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:23.633 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:23.633 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_20 2024-03-11T23:10:23.633 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_20.status 2024-03-11T23:10:23.633 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-03-11T23:10:23.633 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:10:23.678 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:10:23.678 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:10:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:10:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:10:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:10:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:10:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 124 = '' 2024-03-11T23:10:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:10:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:10:24.481 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_20.status 2024-03-11T23:10:24.481 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:24.481 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:24.481 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_20 2024-03-11T23:10:24.481 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_20.status 2024-03-11T23:10:24.481 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-03-11T23:10:24.481 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:10:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:10:24.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=124 2024-03-11T23:10:24.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 124 = 124 2024-03-11T23:10:24.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:10:24.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2024-03-11T23:10:24.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:24.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:24.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_20 2024-03-11T23:10:24.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:10:24.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:10:24.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:10:24.528 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:10:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2024-03-11T23:10:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_20': 2024-03-11T23:10:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:10:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:10:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:10:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 1288d56f7bfc 2024-03-11T23:10:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.1288d56f7bfc 2024-03-11T23:10:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:10:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:10:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:10:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:10:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:02:39 2024 2024-03-11T23:10:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:02:39 2024 2024-03-11T23:10:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:02:39 2024 2024-03-11T23:10:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:10:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:10:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:10:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 7b95035b-6c99-40a9-947e-f623904f126b 2024-03-11T23:10:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:10:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:10:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_20 snap 2024-03-11T23:10:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_20 2024-03-11T23:10:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:10:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:10:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_20.export 2024-03-11T23:10:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_20.export 2024-03-11T23:10:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_20.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_20.export 2024-03-11T23:10:25.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_20.export 2024-03-11T23:10:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:10:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_20.export 2024-03-11T23:10:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:10:26.195 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_20.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_20.export 2024-03-11T23:10:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_20.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_20.export 2024-03-11T23:10:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:10:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:10:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_21 2024-03-11T23:10:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_21 snap 2024-03-11T23:10:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:10:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_21 2024-03-11T23:10:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:10:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:10:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2024-03-11T23:10:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_21 2024-03-11T23:10:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2024-03-11T23:10:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:26.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:26.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_21 2024-03-11T23:10:26.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:10:26.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:10:26.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:10:26.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:10:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2024-03-11T23:10:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_21 2024-03-11T23:10:27.931 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:10:27.931 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:10:27.931 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:10:27.931 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2024-03-11T23:10:27.932 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:10:27.932 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:10:27.932 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:27.932 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:10:27.932 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:10:27.932 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:10:27.932 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:10:27.932 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:10:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:10:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:10:27.934 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:10:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2024-03-11T23:10:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:10:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:10:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:10:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:10:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:10:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:10:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:10:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:10:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:10:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:10:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:10:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:10:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:10:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:10:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:10:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:10:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:10:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:10:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:10:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:10:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:10:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:10:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:10:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:10:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:10:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:10:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:10:27.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:10:27.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:10:27.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:10:27.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2024-03-11T23:10:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-03-11T23:10:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:10:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:10:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:10:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_21 2024-03-11T23:10:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:10:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:10:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:10:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:10:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:10:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:10:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:10:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:10:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:10:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:10:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:10:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:10:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:10:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:10:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:10:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:10:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:10:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:10:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:10:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:10:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:10:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:10:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:10:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:10:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:10:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:10:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:10:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:10:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:10:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:10:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_21 2024-03-11T23:10:28.153 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-03-11T23:10:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:10:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:10:28.157 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:10:28.157 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_21 2024-03-11T23:10:28.157 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T23:10:28.157 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:28.157 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:10:28.157 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T23:10:28.157 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T23:10:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:10:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:10:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:10:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T23:10:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T23:10:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:10:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T23:10:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:10:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T23:10:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T23:10:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:10:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:10:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T23:10:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T23:10:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:10:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:10:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:10:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T23:10:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:10:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:10:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T23:10:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T23:10:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T23:10:28.175 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:10:28.175 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:10:28.175 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:10:28.175 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_21 2024-03-11T23:10:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-03-11T23:10:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:10:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:10:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:10:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_21 2024-03-11T23:10:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:3 2024-03-11T23:10:28.263 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:28.263 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:10:28.263 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-03-11T23:10:28.263 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:3 2024-03-11T23:10:28.263 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:10:28.263 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:10:28.263 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:10:28.263 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=3 2024-03-11T23:10:28.263 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 3 = cluster1:3 ']' 2024-03-11T23:10:28.263 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:10:28.263 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:28.263 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=3 2024-03-11T23:10:28.263 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:10:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-03-11T23:10:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:3 2024-03-11T23:10:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:10:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:10:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-03-11T23:10:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:3 2024-03-11T23:10:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:10:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:10:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:10:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 3 = cluster1:3 ']' 2024-03-11T23:10:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:10:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:10:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=3 2024-03-11T23:10:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=3 2024-03-11T23:10:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-03-11T23:10:28.281 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:10:28.281 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:10:28.281 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:10:28.281 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_21 2024-03-11T23:10:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:10:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:10:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:10:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:10:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:10:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:10:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:10:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:10:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2024-03-11T23:10:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:10:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:10:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_21 2024-03-11T23:10:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:10:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:10:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_21 2024-03-11T23:10:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:10:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:10:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_21 2024-03-11T23:10:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_21.status 2024-03-11T23:10:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_21.status 2024-03-11T23:10:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:10:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_21.status 2024-03-11T23:10:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_21.status 2024-03-11T23:10:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_21.status 2024-03-11T23:10:28.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:10:28.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_21.status 2024-03-11T23:10:28.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_21 2024-03-11T23:10:28.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:10:28.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:28.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_21 2024-03-11T23:10:28.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_21 2024-03-11T23:10:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 126 2024-03-11T23:10:28.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_21 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_21.status 2024-03-11T23:10:28.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:10:28.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:28.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_21 2024-03-11T23:10:28.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_21.status 2024-03-11T23:10:28.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_21 --format xml 2024-03-11T23:10:28.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:10:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:10:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=126 2024-03-11T23:10:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:10:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:10:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:10:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_21.status 2024-03-11T23:10:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_21 2024-03-11T23:10:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_21.status 2024-03-11T23:10:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-03-11T23:10:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:10:29.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:10:29.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:10:29.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:10:29.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:10:29.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:10:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:10:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 126 = '' 2024-03-11T23:10:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:10:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:10:29.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_21.status 2024-03-11T23:10:29.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:29.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:29.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_21 2024-03-11T23:10:29.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_21.status 2024-03-11T23:10:29.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-03-11T23:10:29.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:10:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:10:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:10:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:10:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:10:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:10:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:10:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 126 = '' 2024-03-11T23:10:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:10:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:10:30.494 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_21.status 2024-03-11T23:10:30.494 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:30.494 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:30.494 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_21 2024-03-11T23:10:30.494 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_21.status 2024-03-11T23:10:30.494 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-03-11T23:10:30.494 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:10:30.539 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:10:30.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=126 2024-03-11T23:10:30.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 126 = 126 2024-03-11T23:10:30.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:10:30.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2024-03-11T23:10:30.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:30.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:30.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_21 2024-03-11T23:10:30.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:10:30.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:10:30.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:10:30.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:10:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2024-03-11T23:10:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_21': 2024-03-11T23:10:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:10:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:10:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:10:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 10b3afded721 2024-03-11T23:10:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.10b3afded721 2024-03-11T23:10:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:10:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:10:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:10:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:10:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:02:45 2024 2024-03-11T23:10:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:02:45 2024 2024-03-11T23:10:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:02:45 2024 2024-03-11T23:10:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:10:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:10:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:10:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 32652090-5ba2-4262-a3f3-521e042b1019 2024-03-11T23:10:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:10:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:10:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_21 snap 2024-03-11T23:10:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_21 2024-03-11T23:10:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:10:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:10:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_21.export 2024-03-11T23:10:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_21.export 2024-03-11T23:10:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_21.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_21.export 2024-03-11T23:10:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_21.export 2024-03-11T23:10:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:10:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_21.export 2024-03-11T23:10:32.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:10:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_21.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_21.export 2024-03-11T23:10:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_21.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_21.export 2024-03-11T23:10:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:10:32.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:10:32.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_22 2024-03-11T23:10:32.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_22 snap 2024-03-11T23:10:32.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:10:32.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:32.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_22 2024-03-11T23:10:32.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:10:32.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:10:32.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2024-03-11T23:10:32.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:32.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:32.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_22 2024-03-11T23:10:32.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2024-03-11T23:10:32.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:32.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:32.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_22 2024-03-11T23:10:32.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:10:32.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:10:32.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:10:32.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:10:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2024-03-11T23:10:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_22 2024-03-11T23:10:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:10:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:10:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:10:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2024-03-11T23:10:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:10:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:10:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:10:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:10:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:10:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:10:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:10:33.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:10:33.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:10:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:10:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2024-03-11T23:10:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:10:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:10:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:10:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:10:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:10:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:10:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:10:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:10:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:10:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:10:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:10:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:10:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:10:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:10:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:10:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:10:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:10:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:10:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:10:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:10:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:10:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:10:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:10:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:10:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:10:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:10:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:10:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:10:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:10:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:10:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2024-03-11T23:10:34.065 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-03-11T23:10:34.065 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:10:34.065 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:10:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:10:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_22 2024-03-11T23:10:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:10:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:10:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:10:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:10:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:10:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:10:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:10:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:10:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:10:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:10:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:10:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:10:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:10:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:10:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:10:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:10:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:10:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:10:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:10:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:10:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:10:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:10:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:10:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:10:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:10:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:10:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:10:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:10:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:10:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:10:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_22 2024-03-11T23:10:34.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-03-11T23:10:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:10:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:10:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:10:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_22 2024-03-11T23:10:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T23:10:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:10:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T23:10:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T23:10:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:10:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:10:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:10:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T23:10:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T23:10:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:10:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T23:10:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:10:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T23:10:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T23:10:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:10:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:10:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T23:10:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T23:10:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:10:34.174 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:10:34.174 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:34.174 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:10:34.174 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T23:10:34.174 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:10:34.174 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:10:34.174 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T23:10:34.174 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T23:10:34.174 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T23:10:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:10:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:10:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:10:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_22 2024-03-11T23:10:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-03-11T23:10:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:10:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:10:34.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:10:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_22 2024-03-11T23:10:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:3 2024-03-11T23:10:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:10:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-03-11T23:10:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:3 2024-03-11T23:10:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:10:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:10:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:10:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=3 2024-03-11T23:10:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 3 = cluster1:3 ']' 2024-03-11T23:10:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:10:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=3 2024-03-11T23:10:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:10:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-03-11T23:10:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:3 2024-03-11T23:10:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:10:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:10:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-03-11T23:10:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:3 2024-03-11T23:10:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:10:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:10:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:10:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 3 = cluster1:3 ']' 2024-03-11T23:10:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:10:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:10:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=3 2024-03-11T23:10:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=3 2024-03-11T23:10:34.279 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-03-11T23:10:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:10:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:10:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:10:34.294 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_22 2024-03-11T23:10:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:10:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:10:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:10:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:10:34.384 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:10:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:10:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:10:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:10:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2024-03-11T23:10:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:10:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:10:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_22 2024-03-11T23:10:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:10:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:10:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_22 2024-03-11T23:10:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:10:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:10:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_22 2024-03-11T23:10:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_22.status 2024-03-11T23:10:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_22.status 2024-03-11T23:10:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:10:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_22.status 2024-03-11T23:10:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_22.status 2024-03-11T23:10:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_22.status 2024-03-11T23:10:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:10:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_22.status 2024-03-11T23:10:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_22 2024-03-11T23:10:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:10:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_22 2024-03-11T23:10:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_22 2024-03-11T23:10:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 128 2024-03-11T23:10:34.957 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_22 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_22.status 2024-03-11T23:10:34.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:10:34.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:34.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_22 2024-03-11T23:10:34.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_22.status 2024-03-11T23:10:34.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_22 --format xml 2024-03-11T23:10:34.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:10:35.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:10:35.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=128 2024-03-11T23:10:35.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:10:35.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:10:35.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:10:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_22.status 2024-03-11T23:10:35.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:35.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:35.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_22 2024-03-11T23:10:35.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_22.status 2024-03-11T23:10:35.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-03-11T23:10:35.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:10:35.251 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:10:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:10:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:10:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:10:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:10:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:10:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 128 = '' 2024-03-11T23:10:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:10:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:10:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_22.status 2024-03-11T23:10:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_22 2024-03-11T23:10:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_22.status 2024-03-11T23:10:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-03-11T23:10:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:10:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:10:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:10:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:10:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:10:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:10:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:10:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 128 = '' 2024-03-11T23:10:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:10:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:10:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_22.status 2024-03-11T23:10:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_22 2024-03-11T23:10:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_22.status 2024-03-11T23:10:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-03-11T23:10:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:10:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:10:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=128 2024-03-11T23:10:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 128 = 128 2024-03-11T23:10:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:10:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2024-03-11T23:10:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_22 2024-03-11T23:10:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:10:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:10:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:10:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:10:36.824 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8066576488127162) > weight (0.25) 2024-03-11T23:10:36.824 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7673627620772872) > weight (0.25) 2024-03-11T23:10:36.824 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5358260868621036) > weight (0.25) 2024-03-11T23:10:36.824 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-03-11T23:10:36.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-03-11T23:10:36.825 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-03-11T23:10:36.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi132.stderr:2024-03-11T23:10:36.822+0000 7f7d5c75d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 55842) UID: 1000 2024-03-11T23:10:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2024-03-11T23:10:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_22': 2024-03-11T23:10:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:10:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:10:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:10:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 11ff23853367 2024-03-11T23:10:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.11ff23853367 2024-03-11T23:10:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:10:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:10:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:10:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:10:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:02:53 2024 2024-03-11T23:10:37.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:02:53 2024 2024-03-11T23:10:37.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:02:53 2024 2024-03-11T23:10:37.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:10:37.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:10:37.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:10:37.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 6b2da8c4-e5c3-4ec9-acc6-083f710996ff 2024-03-11T23:10:37.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:10:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:10:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_22 snap 2024-03-11T23:10:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:37.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_22 2024-03-11T23:10:37.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:10:37.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:10:37.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_22.export 2024-03-11T23:10:37.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_22.export 2024-03-11T23:10:37.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_22.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_22.export 2024-03-11T23:10:37.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_22.export 2024-03-11T23:10:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:10:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_22.export 2024-03-11T23:10:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:10:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_22.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_22.export 2024-03-11T23:10:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_22.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_22.export 2024-03-11T23:10:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:10:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:10:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_23 2024-03-11T23:10:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_23 snap 2024-03-11T23:10:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:10:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_23 2024-03-11T23:10:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:10:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:10:38.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2024-03-11T23:10:38.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:38.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:38.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_23 2024-03-11T23:10:38.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2024-03-11T23:10:38.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:38.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:38.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_23 2024-03-11T23:10:38.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:10:38.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:10:38.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:10:38.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:10:39.825 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-03-11T23:10:39.825 INFO:teuthology.orchestra.run:waiting for 600 2024-03-11T23:10:39.826 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-03-11T23:10:39.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-03-11T23:10:39.829 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-03-11T23:10:39.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-03-11T23:10:39.832 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2024-03-11T23:10:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2024-03-11T23:10:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:10:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:10:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_23 2024-03-11T23:10:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:10:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:10:39.966 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:10:39.966 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2024-03-11T23:10:39.966 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:10:39.966 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:10:39.966 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:39.966 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:10:39.966 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:10:39.966 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:10:39.966 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:10:39.966 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:10:39.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:10:39.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:10:39.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:10:39.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2024-03-11T23:10:39.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:10:39.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:39.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:10:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:10:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:10:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:10:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:10:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:10:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:10:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:10:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:10:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:10:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:10:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:10:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:10:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:10:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:10:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:10:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:10:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:10:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:10:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:10:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:10:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:10:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:10:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:10:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:10:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:10:39.988 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:10:39.988 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:10:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:10:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2024-03-11T23:10:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-03-11T23:10:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:10:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:10:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:10:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_23 2024-03-11T23:10:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:10:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:10:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:10:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:10:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:10:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:10:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:10:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:10:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:10:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:10:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:10:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:10:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:10:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:10:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:10:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:10:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:10:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:10:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:10:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:10:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:10:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:10:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:10:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:10:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:10:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:10:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:10:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:10:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:10:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:10:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_23 2024-03-11T23:10:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-03-11T23:10:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:10:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:10:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:10:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_23 2024-03-11T23:10:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T23:10:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:10:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T23:10:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T23:10:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:10:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:10:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:10:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T23:10:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T23:10:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:10:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T23:10:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:10:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T23:10:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T23:10:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:10:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:10:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T23:10:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T23:10:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:10:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:10:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:10:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T23:10:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:10:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:10:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T23:10:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T23:10:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T23:10:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:10:40.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:10:40.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:10:40.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_23 2024-03-11T23:10:40.288 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-03-11T23:10:40.288 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:10:40.288 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:10:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:10:40.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_23 2024-03-11T23:10:40.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:3 2024-03-11T23:10:40.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:40.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:10:40.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-03-11T23:10:40.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:3 2024-03-11T23:10:40.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:10:40.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:10:40.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:10:40.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=3 2024-03-11T23:10:40.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 3 = cluster1:3 ']' 2024-03-11T23:10:40.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:10:40.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:40.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=3 2024-03-11T23:10:40.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:10:40.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-03-11T23:10:40.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:3 2024-03-11T23:10:40.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:10:40.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:10:40.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-03-11T23:10:40.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:3 2024-03-11T23:10:40.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:10:40.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:10:40.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:40.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:10:40.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 3 = cluster1:3 ']' 2024-03-11T23:10:40.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:10:40.294 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:10:40.294 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=3 2024-03-11T23:10:40.294 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=3 2024-03-11T23:10:40.294 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-03-11T23:10:40.309 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:10:40.309 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:10:40.309 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:10:40.309 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_23 2024-03-11T23:10:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-03-11T23:10:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:10:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:10:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:10:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 1 2024-03-11T23:10:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:10:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:10:41.388 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:10:41.388 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2024-03-11T23:10:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:10:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:10:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:10:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:10:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:10:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:10:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:10:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:10:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:10:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:10:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:10:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:10:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:10:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:10:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:10:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:10:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:10:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:10:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:10:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:10:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:10:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:10:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:10:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:10:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:10:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:10:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:10:41.409 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:10:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:10:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:10:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2024-03-11T23:10:41.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-03-11T23:10:41.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:10:41.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:10:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:10:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_23 2024-03-11T23:10:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:10:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:10:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:10:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:10:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:10:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:10:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:10:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:10:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:10:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:10:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:10:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:10:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:10:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:10:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:10:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:10:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:10:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:10:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:10:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:10:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:10:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:10:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:10:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:10:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:10:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:10:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:10:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:10:41.521 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:10:41.521 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:10:41.521 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_23 2024-03-11T23:10:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-03-11T23:10:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:10:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:10:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:10:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_23 2024-03-11T23:10:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T23:10:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:10:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T23:10:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T23:10:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:10:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:10:41.610 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:10:41.610 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T23:10:41.610 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T23:10:41.610 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:10:41.610 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:41.610 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T23:10:41.610 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:10:41.610 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T23:10:41.610 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T23:10:41.610 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:10:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:10:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T23:10:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T23:10:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:10:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:10:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:10:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T23:10:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:10:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:10:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T23:10:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T23:10:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T23:10:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:10:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:10:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:10:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_23 2024-03-11T23:10:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-03-11T23:10:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:10:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:10:41.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:10:41.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_23 2024-03-11T23:10:41.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:3 2024-03-11T23:10:41.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:41.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:10:41.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-03-11T23:10:41.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:3 2024-03-11T23:10:41.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:10:41.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:10:41.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:10:41.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=3 2024-03-11T23:10:41.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 3 = cluster1:3 ']' 2024-03-11T23:10:41.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:10:41.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:41.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=3 2024-03-11T23:10:41.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:10:41.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-03-11T23:10:41.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:3 2024-03-11T23:10:41.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:10:41.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:10:41.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-03-11T23:10:41.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:3 2024-03-11T23:10:41.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:10:41.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:10:41.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:41.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:10:41.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 3 = cluster1:3 ']' 2024-03-11T23:10:41.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:10:41.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:10:41.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=3 2024-03-11T23:10:41.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=3 2024-03-11T23:10:41.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-03-11T23:10:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:10:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:10:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:10:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_23 2024-03-11T23:10:41.806 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-03-11T23:10:41.806 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:10:41.807 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:10:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:10:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 2 2024-03-11T23:10:43.809 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:10:43.809 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:10:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:10:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2024-03-11T23:10:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:10:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:10:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:10:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:10:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:10:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:10:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:10:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:10:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:10:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:10:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:10:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:10:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:10:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:10:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:10:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:10:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:10:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:10:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:10:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:10:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:43.815 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:10:43.815 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:10:43.815 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:10:43.815 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:10:43.815 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:10:43.815 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:10:43.815 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:10:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:10:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:10:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:10:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2024-03-11T23:10:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-03-11T23:10:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:10:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:10:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:10:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_23 2024-03-11T23:10:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:10:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:10:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:10:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:10:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:10:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:10:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:10:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:10:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:10:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:10:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:10:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:10:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:10:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:10:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:10:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:10:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:10:43.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:10:43.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:10:43.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:10:43.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:43.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:10:43.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:10:43.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:10:43.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:10:43.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:10:43.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:10:43.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:10:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:10:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:10:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:10:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_23 2024-03-11T23:10:44.030 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-03-11T23:10:44.030 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:10:44.030 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:10:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:10:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_23 2024-03-11T23:10:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T23:10:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:10:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T23:10:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T23:10:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:10:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:10:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:10:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T23:10:44.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T23:10:44.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:10:44.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:44.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T23:10:44.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:10:44.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T23:10:44.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T23:10:44.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:10:44.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:10:44.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T23:10:44.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T23:10:44.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:10:44.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:10:44.036 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:44.036 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:10:44.036 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T23:10:44.036 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:10:44.036 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:10:44.036 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T23:10:44.036 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T23:10:44.036 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T23:10:44.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:10:44.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:10:44.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:10:44.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_23 2024-03-11T23:10:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-03-11T23:10:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:10:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:10:44.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:10:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_23 2024-03-11T23:10:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:3 2024-03-11T23:10:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:10:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-03-11T23:10:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:3 2024-03-11T23:10:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:10:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:10:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:10:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=3 2024-03-11T23:10:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 3 = cluster1:3 ']' 2024-03-11T23:10:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:10:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=3 2024-03-11T23:10:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:10:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-03-11T23:10:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:3 2024-03-11T23:10:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:10:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:10:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-03-11T23:10:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:3 2024-03-11T23:10:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:10:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:10:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:10:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 3 = cluster1:3 ']' 2024-03-11T23:10:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:10:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:10:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=3 2024-03-11T23:10:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=3 2024-03-11T23:10:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-03-11T23:10:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:10:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:10:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:10:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_23 2024-03-11T23:10:44.229 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-03-11T23:10:44.229 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:10:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:10:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:10:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 4 2024-03-11T23:10:48.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:10:48.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:10:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:10:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2024-03-11T23:10:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:10:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:10:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:10:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:10:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:10:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:10:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:10:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:10:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:10:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:10:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:10:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:10:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:10:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:10:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:10:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:10:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:10:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:10:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:10:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:10:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:10:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:10:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:10:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:10:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:10:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:10:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:10:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:10:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:10:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:10:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2024-03-11T23:10:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-03-11T23:10:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:10:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:10:48.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:10:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_23 2024-03-11T23:10:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:10:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:10:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:10:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:10:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:10:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:10:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:10:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:10:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:10:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:10:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:10:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:10:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:10:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:10:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:10:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:10:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:10:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:10:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:10:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:10:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:10:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:10:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:10:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:10:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:10:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:10:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:10:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:10:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:10:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:10:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_23 2024-03-11T23:10:48.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-03-11T23:10:48.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:10:48.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:10:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:10:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_23 2024-03-11T23:10:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T23:10:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:10:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T23:10:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T23:10:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:10:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:10:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:10:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T23:10:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T23:10:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:10:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T23:10:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:10:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T23:10:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T23:10:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:10:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:10:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T23:10:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T23:10:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:10:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:10:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:10:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T23:10:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:10:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:10:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T23:10:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T23:10:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T23:10:48.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:10:48.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:10:48.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:10:48.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_23 2024-03-11T23:10:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-03-11T23:10:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:10:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:10:48.557 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:10:48.557 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_23 2024-03-11T23:10:48.557 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:3 2024-03-11T23:10:48.557 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:48.557 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:10:48.557 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-03-11T23:10:48.557 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:3 2024-03-11T23:10:48.557 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:10:48.557 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:10:48.557 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:10:48.557 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=3 2024-03-11T23:10:48.557 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 3 = cluster1:3 ']' 2024-03-11T23:10:48.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:10:48.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:48.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=3 2024-03-11T23:10:48.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:10:48.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-03-11T23:10:48.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:3 2024-03-11T23:10:48.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:10:48.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:10:48.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-03-11T23:10:48.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:3 2024-03-11T23:10:48.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:10:48.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:10:48.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:48.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:10:48.559 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 3 = cluster1:3 ']' 2024-03-11T23:10:48.559 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:10:48.559 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:10:48.559 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=3 2024-03-11T23:10:48.559 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=3 2024-03-11T23:10:48.559 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-03-11T23:10:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:10:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:10:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:10:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_23 2024-03-11T23:10:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-03-11T23:10:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:10:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:10:48.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:10:48.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 8 2024-03-11T23:10:56.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:10:56.655 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:10:56.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:10:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2024-03-11T23:10:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:10:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:10:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:10:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:10:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:10:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:10:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:10:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:10:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:10:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:10:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:10:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:10:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:10:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:10:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:10:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:10:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:10:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:10:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:10:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:10:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:10:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:10:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:10:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:10:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:10:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:10:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:10:56.679 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:10:56.679 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:10:56.679 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:10:56.679 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2024-03-11T23:10:56.767 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-03-11T23:10:56.768 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:10:56.768 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:10:56.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:10:56.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_23 2024-03-11T23:10:56.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:10:56.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:56.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:10:56.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:10:56.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:10:56.772 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:10:56.772 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:10:56.772 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:10:56.772 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:10:56.772 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:10:56.772 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:10:56.772 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:56.772 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:10:56.772 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:10:56.772 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:10:56.772 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:10:56.772 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:10:56.772 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:10:56.772 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:10:56.772 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:10:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:10:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:10:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:10:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:10:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:10:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:10:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:10:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:10:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:10:56.789 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:10:56.789 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:10:56.789 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:10:56.789 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_23 2024-03-11T23:10:56.870 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-03-11T23:10:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:10:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:10:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:10:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_23 2024-03-11T23:10:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T23:10:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:56.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:10:56.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T23:10:56.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T23:10:56.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:10:56.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:10:56.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:10:56.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T23:10:56.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T23:10:56.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:10:56.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:56.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T23:10:56.876 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:10:56.876 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T23:10:56.876 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T23:10:56.876 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:10:56.876 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:10:56.876 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T23:10:56.876 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T23:10:56.876 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:10:56.876 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:10:56.876 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:56.876 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:10:56.876 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T23:10:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:10:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:10:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T23:10:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T23:10:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T23:10:56.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:10:56.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:10:56.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:10:56.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_23 2024-03-11T23:10:56.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-03-11T23:10:56.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:10:56.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:10:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:10:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_23 2024-03-11T23:10:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:3 2024-03-11T23:10:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:10:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:10:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-03-11T23:10:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:3 2024-03-11T23:10:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:10:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:10:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:10:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=3 2024-03-11T23:10:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 3 = cluster1:3 ']' 2024-03-11T23:10:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:10:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=3 2024-03-11T23:10:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:10:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-03-11T23:10:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:3 2024-03-11T23:10:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:10:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:10:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-03-11T23:10:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:3 2024-03-11T23:10:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:10:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:10:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:10:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:10:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 3 = cluster1:3 ']' 2024-03-11T23:10:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:10:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:10:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=3 2024-03-11T23:10:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=3 2024-03-11T23:10:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-03-11T23:10:57.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:10:57.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:10:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:10:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_23 2024-03-11T23:10:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-03-11T23:10:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:10:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:10:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:10:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 8 2024-03-11T23:11:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:11:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:11:05.082 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:11:05.082 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2024-03-11T23:11:05.082 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:11:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:11:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:11:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:11:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:11:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:11:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:11:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:11:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:11:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:11:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:11:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:11:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:11:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:11:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:11:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:11:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:11:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:11:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:11:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:11:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:11:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:11:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:11:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:11:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:11:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:11:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:11:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:11:05.103 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:11:05.103 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:11:05.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:11:05.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2024-03-11T23:11:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-03-11T23:11:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:11:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:11:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:11:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_23 2024-03-11T23:11:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:11:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:11:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:11:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:11:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:11:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:11:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:11:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:11:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:11:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:11:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:11:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:11:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:11:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:11:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:11:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:11:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:11:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:11:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:11:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:11:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:11:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:11:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:11:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:11:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:11:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:11:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:11:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:11:05.239 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:11:05.239 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:11:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:11:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_23 2024-03-11T23:11:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-03-11T23:11:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:11:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:11:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:11:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_23 2024-03-11T23:11:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T23:11:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:11:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:11:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T23:11:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T23:11:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:11:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:11:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:11:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T23:11:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T23:11:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:11:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T23:11:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:11:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T23:11:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T23:11:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:11:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:11:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T23:11:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T23:11:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:11:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:11:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:11:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T23:11:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:11:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:11:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T23:11:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T23:11:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T23:11:05.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:11:05.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:11:05.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:11:05.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_23 2024-03-11T23:11:05.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-03-11T23:11:05.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:11:05.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:11:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:11:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_23 2024-03-11T23:11:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:3 2024-03-11T23:11:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:11:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:11:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-03-11T23:11:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:3 2024-03-11T23:11:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:11:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:11:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:11:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=3 2024-03-11T23:11:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 3 = cluster1:3 ']' 2024-03-11T23:11:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:11:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=3 2024-03-11T23:11:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:11:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-03-11T23:11:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:3 2024-03-11T23:11:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:11:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:11:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-03-11T23:11:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:3 2024-03-11T23:11:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:11:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:11:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:11:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 3 = cluster1:3 ']' 2024-03-11T23:11:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:11:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:11:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=3 2024-03-11T23:11:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=3 2024-03-11T23:11:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-03-11T23:11:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:11:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:11:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:11:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_23 2024-03-11T23:11:05.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:11:05.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:11:05.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:11:05.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:11:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:11:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:11:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:11:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:11:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2024-03-11T23:11:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:11:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_23 2024-03-11T23:11:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:11:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:11:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_23 2024-03-11T23:11:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:11:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_23 2024-03-11T23:11:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_23.status 2024-03-11T23:11:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_23.status 2024-03-11T23:11:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:11:05.543 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_23.status 2024-03-11T23:11:05.543 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_23.status 2024-03-11T23:11:05.543 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_23.status 2024-03-11T23:11:05.543 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:11:05.543 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_23.status 2024-03-11T23:11:05.543 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_23 2024-03-11T23:11:05.543 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:05.543 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:05.543 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_23 2024-03-11T23:11:05.543 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_23 2024-03-11T23:11:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 130 2024-03-11T23:11:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_23 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_23.status 2024-03-11T23:11:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_23 2024-03-11T23:11:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_23.status 2024-03-11T23:11:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_23 --format xml 2024-03-11T23:11:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:11:06.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=130 2024-03-11T23:11:06.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:11:06.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:11:06.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:11:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_23.status 2024-03-11T23:11:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_23 2024-03-11T23:11:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_23.status 2024-03-11T23:11:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-03-11T23:11:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:11:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:11:06.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:11:06.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:11:06.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:11:06.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:11:06.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 130 = '' 2024-03-11T23:11:06.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:11:06.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:11:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_23.status 2024-03-11T23:11:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_23 2024-03-11T23:11:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_23.status 2024-03-11T23:11:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-03-11T23:11:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:11:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:11:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:11:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:11:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:11:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:11:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 130 = '' 2024-03-11T23:11:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:11:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:11:07.545 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_23.status 2024-03-11T23:11:07.545 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:07.545 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:07.545 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_23 2024-03-11T23:11:07.545 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_23.status 2024-03-11T23:11:07.545 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-03-11T23:11:07.546 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:07.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:11:07.591 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=130 2024-03-11T23:11:07.591 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 130 = 130 2024-03-11T23:11:07.591 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:11:07.591 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2024-03-11T23:11:07.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:07.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:07.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_23 2024-03-11T23:11:07.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:11:07.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:11:07.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:11:07.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:11:07.920 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8829118410400955) > weight (0.25) 2024-03-11T23:11:07.920 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5621164127270941) > weight (0.25) 2024-03-11T23:11:07.920 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-03-11T23:11:07.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-03-11T23:11:07.921 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-03-11T23:11:07.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi132.stderr:2024-03-11T23:11:07.918+0000 7f0ac9550640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 54720) UID: 1000 2024-03-11T23:11:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2024-03-11T23:11:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_23': 2024-03-11T23:11:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:11:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:11:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:11:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 1260f5ef671b 2024-03-11T23:11:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.1260f5ef671b 2024-03-11T23:11:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:11:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:11:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:11:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:11:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:03:01 2024 2024-03-11T23:11:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:03:01 2024 2024-03-11T23:11:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:03:01 2024 2024-03-11T23:11:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:11:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:11:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:11:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 6f278f19-20af-4e77-81d1-1dfe17bc389b 2024-03-11T23:11:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:11:08.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:11:08.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_23 snap 2024-03-11T23:11:08.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:08.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_23 2024-03-11T23:11:08.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:11:08.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:11:08.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_23.export 2024-03-11T23:11:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_23.export 2024-03-11T23:11:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_23.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_23.export 2024-03-11T23:11:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_23.export 2024-03-11T23:11:08.839 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:11:08.847 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_23.export 2024-03-11T23:11:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:11:09.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_23.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_23.export 2024-03-11T23:11:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_23.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_23.export 2024-03-11T23:11:09.184 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:11:09.184 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:11:09.184 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_24 2024-03-11T23:11:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_24 snap 2024-03-11T23:11:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_24 2024-03-11T23:11:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:11:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:11:09.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2024-03-11T23:11:09.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:09.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:09.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_24 2024-03-11T23:11:09.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2024-03-11T23:11:09.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:09.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:09.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_24 2024-03-11T23:11:09.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:11:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:11:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:11:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:11:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2024-03-11T23:11:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_24 2024-03-11T23:11:10.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:11:10.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:11:10.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:11:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2024-03-11T23:11:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:11:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:11:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:11:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:11:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:11:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:11:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:11:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:11:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:11:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:11:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:11:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2024-03-11T23:11:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:11:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:11:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:11:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:11:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:11:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:11:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:11:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:11:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:11:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:11:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:11:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:11:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:11:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:11:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:11:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:11:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:11:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:11:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:11:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:11:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:11:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:11:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:11:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:11:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:11:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:11:11.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:11:11.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:11:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:11:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:11:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:11:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2024-03-11T23:11:11.113 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-03-11T23:11:11.113 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:11:11.113 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:11:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:11:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2024-03-11T23:11:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:11:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:11:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:11:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:11:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:11:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:11:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:11:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:11:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:11:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:11:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:11:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:11:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:11:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:11:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:11:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:11:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:11:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:11:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:11:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:11:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:11:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:11:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:11:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:11:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:11:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:11:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:11:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:11:11.134 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:11:11.134 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:11:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:11:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2024-03-11T23:11:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:11:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:11:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:11:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:11:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:11:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:11:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:11:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:11:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2024-03-11T23:11:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:11:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_24 2024-03-11T23:11:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:11:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:11:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_24 2024-03-11T23:11:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:11:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_24 2024-03-11T23:11:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_24.status 2024-03-11T23:11:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_24.status 2024-03-11T23:11:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:11:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_24.status 2024-03-11T23:11:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_24.status 2024-03-11T23:11:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_24.status 2024-03-11T23:11:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:11:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_24.status 2024-03-11T23:11:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_24 2024-03-11T23:11:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_24 2024-03-11T23:11:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_24 2024-03-11T23:11:12.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 132 2024-03-11T23:11:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_24 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_24.status 2024-03-11T23:11:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_24 2024-03-11T23:11:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_24.status 2024-03-11T23:11:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_24 --format xml 2024-03-11T23:11:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:11:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=132 2024-03-11T23:11:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:11:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:11:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:11:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_24.status 2024-03-11T23:11:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_24 2024-03-11T23:11:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_24.status 2024-03-11T23:11:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-03-11T23:11:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:11:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:11:12.310 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:11:12.310 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:11:12.310 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:11:12.310 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:11:12.310 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 132 = '' 2024-03-11T23:11:12.310 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:11:12.310 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:11:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_24.status 2024-03-11T23:11:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_24 2024-03-11T23:11:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_24.status 2024-03-11T23:11:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-03-11T23:11:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:11:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:11:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:11:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:11:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:11:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:11:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 132 = '' 2024-03-11T23:11:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:11:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:11:12.922 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-03-11T23:11:12.922 INFO:teuthology.orchestra.run:waiting for 600 2024-03-11T23:11:12.922 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-03-11T23:11:12.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-03-11T23:11:12.923 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-03-11T23:11:12.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-03-11T23:11:12.925 INFO:tasks.rbd_mirror_thrash:waiting for 81 secs before thrashing 2024-03-11T23:11:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_24.status 2024-03-11T23:11:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_24 2024-03-11T23:11:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_24.status 2024-03-11T23:11:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-03-11T23:11:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:13.609 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:11:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=132 2024-03-11T23:11:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 132 = 132 2024-03-11T23:11:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:11:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2024-03-11T23:11:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_24 2024-03-11T23:11:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:11:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:11:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:11:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:11:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2024-03-11T23:11:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_24': 2024-03-11T23:11:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:11:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:11:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:11:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 1288d3219463 2024-03-11T23:11:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.1288d3219463 2024-03-11T23:11:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:11:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:11:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:11:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:11:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:03:13 2024 2024-03-11T23:11:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:03:13 2024 2024-03-11T23:11:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:03:13 2024 2024-03-11T23:11:14.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:11:14.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:11:14.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:11:14.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: edbba68c-29ef-4285-9e1b-327ac0002f9c 2024-03-11T23:11:14.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:11:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:11:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_24 snap 2024-03-11T23:11:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_24 2024-03-11T23:11:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:11:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:11:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_24.export 2024-03-11T23:11:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_24.export 2024-03-11T23:11:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_24.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_24.export 2024-03-11T23:11:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_24.export 2024-03-11T23:11:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:11:14.856 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_24.export 2024-03-11T23:11:15.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:11:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_24.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_24.export 2024-03-11T23:11:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_24.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_24.export 2024-03-11T23:11:15.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:11:15.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:11:15.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_25 2024-03-11T23:11:15.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_25 snap 2024-03-11T23:11:15.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:15.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:15.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_25 2024-03-11T23:11:15.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:11:15.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:11:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2024-03-11T23:11:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_25 2024-03-11T23:11:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2024-03-11T23:11:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_25 2024-03-11T23:11:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:11:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:11:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:11:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:11:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2024-03-11T23:11:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_25 2024-03-11T23:11:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:11:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:11:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:11:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2024-03-11T23:11:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:11:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:11:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:11:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:11:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:11:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:11:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:11:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:11:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:11:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:11:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:11:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2024-03-11T23:11:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:11:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:11:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:11:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:11:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:11:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:11:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:11:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:11:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:11:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:11:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:11:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:11:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:11:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:11:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:11:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:11:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:11:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:11:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:11:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:11:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:11:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:11:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:11:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:11:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:11:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:11:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:11:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:11:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:11:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:11:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:11:17.039 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2024-03-11T23:11:17.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-03-11T23:11:17.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:11:17.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:11:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:11:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_25 2024-03-11T23:11:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:11:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:11:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:11:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:11:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:11:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:11:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:11:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:11:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:11:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:11:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:11:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:11:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:11:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:11:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:11:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:11:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:11:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:11:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:11:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:11:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:11:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:11:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:11:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:11:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:11:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:11:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:11:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:11:17.154 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:11:17.154 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:11:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:11:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_25 2024-03-11T23:11:17.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-03-11T23:11:17.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:11:17.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:11:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:11:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_25 2024-03-11T23:11:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T23:11:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:11:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:11:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T23:11:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T23:11:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:11:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:11:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:11:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T23:11:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T23:11:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:11:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T23:11:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:11:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T23:11:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T23:11:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:11:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:11:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T23:11:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T23:11:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:11:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:11:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:11:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T23:11:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:11:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:11:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T23:11:17.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T23:11:17.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T23:11:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:11:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:11:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:11:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_25 2024-03-11T23:11:17.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-03-11T23:11:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:11:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:11:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:11:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_25 2024-03-11T23:11:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:3 2024-03-11T23:11:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:11:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:11:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-03-11T23:11:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:3 2024-03-11T23:11:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:11:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:11:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:11:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=3 2024-03-11T23:11:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 3 = cluster1:3 ']' 2024-03-11T23:11:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:11:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=3 2024-03-11T23:11:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:11:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-03-11T23:11:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:3 2024-03-11T23:11:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:11:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:11:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-03-11T23:11:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:3 2024-03-11T23:11:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:11:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:11:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:11:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 3 = cluster1:3 ']' 2024-03-11T23:11:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:11:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:11:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=3 2024-03-11T23:11:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=3 2024-03-11T23:11:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-03-11T23:11:17.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:11:17.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:11:17.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:11:17.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_25 2024-03-11T23:11:17.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:11:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:11:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:11:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:11:17.456 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:11:17.457 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:11:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:11:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:11:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2024-03-11T23:11:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:11:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_25 2024-03-11T23:11:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:11:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:11:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_25 2024-03-11T23:11:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:11:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_25 2024-03-11T23:11:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_25.status 2024-03-11T23:11:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_25.status 2024-03-11T23:11:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:11:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_25.status 2024-03-11T23:11:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_25.status 2024-03-11T23:11:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_25.status 2024-03-11T23:11:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:11:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_25.status 2024-03-11T23:11:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_25 2024-03-11T23:11:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_25 2024-03-11T23:11:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_25 2024-03-11T23:11:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 134 2024-03-11T23:11:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_25 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_25.status 2024-03-11T23:11:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_25 2024-03-11T23:11:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_25.status 2024-03-11T23:11:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_25 --format xml 2024-03-11T23:11:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:18.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:11:18.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=134 2024-03-11T23:11:18.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:11:18.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:11:18.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:11:18.265 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_25.status 2024-03-11T23:11:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_25 2024-03-11T23:11:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_25.status 2024-03-11T23:11:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-03-11T23:11:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:11:18.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=59 2024-03-11T23:11:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 134 = 59 2024-03-11T23:11:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:11:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:11:18.720 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_25.status 2024-03-11T23:11:18.720 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:18.720 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:18.720 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_25 2024-03-11T23:11:18.720 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_25.status 2024-03-11T23:11:18.720 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-03-11T23:11:18.720 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:18.766 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:11:18.768 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=59 2024-03-11T23:11:18.768 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 134 = 59 2024-03-11T23:11:18.768 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:11:18.768 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:11:19.570 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_25.status 2024-03-11T23:11:19.570 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:19.570 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:19.570 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_25 2024-03-11T23:11:19.570 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_25.status 2024-03-11T23:11:19.570 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-03-11T23:11:19.570 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:11:19.622 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:11:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:11:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:11:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:11:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:11:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 134 = '' 2024-03-11T23:11:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:11:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:11:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_25.status 2024-03-11T23:11:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_25 2024-03-11T23:11:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_25.status 2024-03-11T23:11:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-03-11T23:11:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:11:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=134 2024-03-11T23:11:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 134 = 134 2024-03-11T23:11:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:11:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2024-03-11T23:11:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_25 2024-03-11T23:11:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:11:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:11:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:11:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:11:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2024-03-11T23:11:22.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_25': 2024-03-11T23:11:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:11:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:11:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:11:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 10b3ec4ece2b 2024-03-11T23:11:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.10b3ec4ece2b 2024-03-11T23:11:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:11:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:11:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:11:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:11:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:03:22 2024 2024-03-11T23:11:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:03:22 2024 2024-03-11T23:11:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:03:22 2024 2024-03-11T23:11:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:11:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:11:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:11:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: d58404ce-fb8c-4239-ba38-049eab7f80aa 2024-03-11T23:11:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:11:22.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:11:22.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_25 snap 2024-03-11T23:11:22.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:22.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_25 2024-03-11T23:11:22.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:11:22.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:11:22.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_25.export 2024-03-11T23:11:22.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_25.export 2024-03-11T23:11:22.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_25.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_25.export 2024-03-11T23:11:22.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_25.export 2024-03-11T23:11:22.523 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:11:22.531 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_25.export 2024-03-11T23:11:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:11:22.768 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_25.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_25.export 2024-03-11T23:11:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_25.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_25.export 2024-03-11T23:11:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:11:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:11:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_26 2024-03-11T23:11:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_26 snap 2024-03-11T23:11:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_26 2024-03-11T23:11:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:11:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:11:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2024-03-11T23:11:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_26 2024-03-11T23:11:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2024-03-11T23:11:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_26 2024-03-11T23:11:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:11:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:11:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:11:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:11:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2024-03-11T23:11:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_26 2024-03-11T23:11:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:11:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:11:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:11:24.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2024-03-11T23:11:24.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:11:24.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:11:24.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:11:24.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:11:24.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:11:24.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:11:24.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:11:24.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:11:24.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:11:24.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:11:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:11:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2024-03-11T23:11:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:11:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:11:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:11:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:11:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:11:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:11:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:11:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:11:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:11:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:11:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:11:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:11:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:11:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:11:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:11:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:11:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:11:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:11:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:11:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:11:24.028 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:11:24.028 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:24.028 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:11:24.028 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:11:24.028 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:11:24.028 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:11:24.028 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:11:24.028 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:11:24.028 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:11:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:11:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:11:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:11:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2024-03-11T23:11:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:11:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:11:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:11:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:11:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:11:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:11:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:11:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:11:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2024-03-11T23:11:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:11:24.146 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:24.146 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:24.146 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_26 2024-03-11T23:11:24.146 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:11:24.146 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:11:24.146 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_26 2024-03-11T23:11:24.146 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:11:24.146 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:24.146 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:24.146 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_26 2024-03-11T23:11:24.146 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_26.status 2024-03-11T23:11:24.146 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_26.status 2024-03-11T23:11:24.146 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:11:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_26.status 2024-03-11T23:11:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_26.status 2024-03-11T23:11:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_26.status 2024-03-11T23:11:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:11:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_26.status 2024-03-11T23:11:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_26 2024-03-11T23:11:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_26 2024-03-11T23:11:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_26 2024-03-11T23:11:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 136 2024-03-11T23:11:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_26 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_26.status 2024-03-11T23:11:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_26 2024-03-11T23:11:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_26.status 2024-03-11T23:11:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_26 --format xml 2024-03-11T23:11:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:25.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:11:25.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=136 2024-03-11T23:11:25.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:11:25.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:11:25.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:11:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_26.status 2024-03-11T23:11:25.298 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:25.298 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:25.298 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_26 2024-03-11T23:11:25.298 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_26.status 2024-03-11T23:11:25.298 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-03-11T23:11:25.298 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:11:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=60 2024-03-11T23:11:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 136 = 60 2024-03-11T23:11:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:11:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:11:25.745 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_26.status 2024-03-11T23:11:25.745 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:25.745 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:25.745 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_26 2024-03-11T23:11:25.745 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_26.status 2024-03-11T23:11:25.745 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-03-11T23:11:25.745 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:11:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=60 2024-03-11T23:11:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 136 = 60 2024-03-11T23:11:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:11:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:11:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_26.status 2024-03-11T23:11:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_26 2024-03-11T23:11:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_26.status 2024-03-11T23:11:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-03-11T23:11:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:11:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:11:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:11:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:11:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:11:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:11:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 136 = '' 2024-03-11T23:11:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:11:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:11:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_26.status 2024-03-11T23:11:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_26 2024-03-11T23:11:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_26.status 2024-03-11T23:11:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-03-11T23:11:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:11:28.303 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=136 2024-03-11T23:11:28.303 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 136 = 136 2024-03-11T23:11:28.303 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:11:28.303 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2024-03-11T23:11:28.303 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:28.303 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:28.303 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_26 2024-03-11T23:11:28.303 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:11:28.303 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:11:28.304 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:11:28.304 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:11:29.304 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2024-03-11T23:11:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_26': 2024-03-11T23:11:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:11:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:11:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:11:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 1297f1fd8734 2024-03-11T23:11:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.1297f1fd8734 2024-03-11T23:11:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:11:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:11:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:11:29.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:11:29.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:03:34 2024 2024-03-11T23:11:29.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:03:34 2024 2024-03-11T23:11:29.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:03:34 2024 2024-03-11T23:11:29.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:11:29.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:11:29.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:11:29.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: d759a618-182e-4efe-82fa-def08c221d57 2024-03-11T23:11:29.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:11:29.353 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:11:29.353 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_26 snap 2024-03-11T23:11:29.353 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:29.353 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_26 2024-03-11T23:11:29.353 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:11:29.353 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:11:29.353 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_26.export 2024-03-11T23:11:29.353 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_26.export 2024-03-11T23:11:29.353 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_26.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_26.export 2024-03-11T23:11:29.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_26.export 2024-03-11T23:11:29.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:11:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_26.export 2024-03-11T23:11:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:11:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_26.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_26.export 2024-03-11T23:11:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_26.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_26.export 2024-03-11T23:11:29.860 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:11:29.860 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:11:29.860 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_27 2024-03-11T23:11:29.860 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_27 snap 2024-03-11T23:11:29.860 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:29.860 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:29.860 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_27 2024-03-11T23:11:29.860 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:11:29.860 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:11:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2024-03-11T23:11:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_27 2024-03-11T23:11:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2024-03-11T23:11:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_27 2024-03-11T23:11:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:11:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:11:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:11:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:11:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2024-03-11T23:11:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_27 2024-03-11T23:11:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:11:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:11:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:11:31.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2024-03-11T23:11:31.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:11:31.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:11:31.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:11:31.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:11:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:11:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:11:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:11:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:11:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:11:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:11:31.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:11:31.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2024-03-11T23:11:31.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:11:31.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:11:31.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:11:31.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:11:31.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:11:31.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:11:31.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:11:31.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:11:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:11:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:11:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:11:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:11:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:11:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:11:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:11:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:11:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:11:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:11:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:11:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:11:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:11:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:11:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:11:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:11:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:11:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:11:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:11:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:11:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:11:31.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:11:31.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:11:31.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2024-03-11T23:11:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-03-11T23:11:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:11:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:11:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:11:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_27 2024-03-11T23:11:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:11:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:11:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:11:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:11:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:11:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:11:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:11:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:11:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:11:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:11:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:11:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:11:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:11:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:11:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:11:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:11:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:11:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:11:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:11:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:11:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:11:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:11:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:11:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:11:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:11:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:11:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:11:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:11:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:11:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:11:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:11:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_27 2024-03-11T23:11:31.263 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-03-11T23:11:31.263 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:11:31.263 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:11:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:11:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_27 2024-03-11T23:11:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T23:11:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:11:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:11:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T23:11:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T23:11:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:11:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:11:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:11:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T23:11:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T23:11:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:11:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T23:11:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:11:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T23:11:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T23:11:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:11:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:11:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T23:11:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T23:11:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:11:31.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:11:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:11:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T23:11:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:11:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:11:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T23:11:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T23:11:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T23:11:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:11:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:11:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:11:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_27 2024-03-11T23:11:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:11:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:11:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:11:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:11:31.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:11:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:11:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:11:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:11:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2024-03-11T23:11:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:11:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_27 2024-03-11T23:11:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:11:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:11:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_27 2024-03-11T23:11:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:11:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_27 2024-03-11T23:11:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_27.status 2024-03-11T23:11:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_27.status 2024-03-11T23:11:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:11:31.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_27.status 2024-03-11T23:11:31.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_27.status 2024-03-11T23:11:31.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_27.status 2024-03-11T23:11:31.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:11:31.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_27.status 2024-03-11T23:11:31.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_27 2024-03-11T23:11:31.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:31.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:31.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_27 2024-03-11T23:11:31.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_27 2024-03-11T23:11:32.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 138 2024-03-11T23:11:32.068 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_27 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_27.status 2024-03-11T23:11:32.068 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_27 2024-03-11T23:11:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_27.status 2024-03-11T23:11:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_27 --format xml 2024-03-11T23:11:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:11:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=138 2024-03-11T23:11:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:11:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:11:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:11:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_27.status 2024-03-11T23:11:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_27 2024-03-11T23:11:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_27.status 2024-03-11T23:11:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-03-11T23:11:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:11:32.572 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=61 2024-03-11T23:11:32.572 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 138 = 61 2024-03-11T23:11:32.572 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:11:32.572 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:11:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_27.status 2024-03-11T23:11:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_27 2024-03-11T23:11:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_27.status 2024-03-11T23:11:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-03-11T23:11:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:11:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:11:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:11:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:11:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:11:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:11:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 138 = '' 2024-03-11T23:11:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:11:33.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:11:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_27.status 2024-03-11T23:11:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_27 2024-03-11T23:11:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_27.status 2024-03-11T23:11:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-03-11T23:11:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:11:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:11:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:11:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:11:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:11:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:11:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 138 = '' 2024-03-11T23:11:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:11:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:11:35.524 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_27.status 2024-03-11T23:11:35.525 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:35.525 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:35.525 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_27 2024-03-11T23:11:35.525 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_27.status 2024-03-11T23:11:35.525 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-03-11T23:11:35.525 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:11:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=138 2024-03-11T23:11:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 138 = 138 2024-03-11T23:11:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:11:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2024-03-11T23:11:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_27 2024-03-11T23:11:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:11:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:11:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:11:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:11:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2024-03-11T23:11:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_27': 2024-03-11T23:11:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:11:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:11:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:11:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 1297fea6830c 2024-03-11T23:11:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.1297fea6830c 2024-03-11T23:11:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:11:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:11:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:11:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:11:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:03:44 2024 2024-03-11T23:11:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:03:44 2024 2024-03-11T23:11:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:03:44 2024 2024-03-11T23:11:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:11:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:11:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:11:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: f55425aa-fe4e-4d48-89cf-caafa49c5423 2024-03-11T23:11:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:11:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:11:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_27 snap 2024-03-11T23:11:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_27 2024-03-11T23:11:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:11:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:11:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_27.export 2024-03-11T23:11:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_27.export 2024-03-11T23:11:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_27.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_27.export 2024-03-11T23:11:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_27.export 2024-03-11T23:11:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:11:36.819 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_27.export 2024-03-11T23:11:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:11:37.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_27.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_27.export 2024-03-11T23:11:37.121 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_27.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_27.export 2024-03-11T23:11:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:11:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:11:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_28 2024-03-11T23:11:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_28 snap 2024-03-11T23:11:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_28 2024-03-11T23:11:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:11:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:11:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2024-03-11T23:11:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_28 2024-03-11T23:11:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2024-03-11T23:11:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_28 2024-03-11T23:11:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:11:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:11:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:11:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:11:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2024-03-11T23:11:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_28 2024-03-11T23:11:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:11:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:11:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:11:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_28 2024-03-11T23:11:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:11:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:11:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:11:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:11:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:11:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:11:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:11:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:11:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:11:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:11:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:11:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2024-03-11T23:11:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:11:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:11:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:11:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:11:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:11:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:11:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:11:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:11:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:11:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:11:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:11:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:11:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:11:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:11:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:11:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:11:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:11:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:11:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:11:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:11:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:11:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:11:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:11:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:11:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:11:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:11:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:11:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:11:39.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:11:39.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:11:39.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:11:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2024-03-11T23:11:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:11:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:11:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:11:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:11:39.198 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:11:39.200 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:11:39.200 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:11:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:11:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2024-03-11T23:11:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:11:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_28 2024-03-11T23:11:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:11:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:11:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_28 2024-03-11T23:11:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:11:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_28 2024-03-11T23:11:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_28.status 2024-03-11T23:11:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_28.status 2024-03-11T23:11:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:11:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_28.status 2024-03-11T23:11:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_28.status 2024-03-11T23:11:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_28.status 2024-03-11T23:11:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:11:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_28.status 2024-03-11T23:11:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_28 2024-03-11T23:11:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_28 2024-03-11T23:11:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_28 2024-03-11T23:11:40.072 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 140 2024-03-11T23:11:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_28 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_28.status 2024-03-11T23:11:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_28 2024-03-11T23:11:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_28.status 2024-03-11T23:11:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_28 --format xml 2024-03-11T23:11:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:11:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=140 2024-03-11T23:11:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:11:40.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:11:40.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:11:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_28.status 2024-03-11T23:11:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_28 2024-03-11T23:11:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_28.status 2024-03-11T23:11:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-03-11T23:11:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:40.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:11:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=62 2024-03-11T23:11:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 140 = 62 2024-03-11T23:11:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:11:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:11:40.777 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_28.status 2024-03-11T23:11:40.777 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:40.777 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:40.777 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_28 2024-03-11T23:11:40.777 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_28.status 2024-03-11T23:11:40.777 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-03-11T23:11:40.777 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:40.826 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:11:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=62 2024-03-11T23:11:40.829 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 140 = 62 2024-03-11T23:11:40.829 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:11:40.829 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:11:41.630 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_28.status 2024-03-11T23:11:41.630 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:41.630 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:41.630 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_28 2024-03-11T23:11:41.630 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_28.status 2024-03-11T23:11:41.630 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-03-11T23:11:41.630 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:11:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:11:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:11:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:11:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:11:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:11:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 140 = '' 2024-03-11T23:11:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:11:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:11:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_28.status 2024-03-11T23:11:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_28 2024-03-11T23:11:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_28.status 2024-03-11T23:11:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-03-11T23:11:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:43.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:11:43.342 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=140 2024-03-11T23:11:43.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 140 = 140 2024-03-11T23:11:43.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:11:43.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2024-03-11T23:11:43.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:43.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:43.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_28 2024-03-11T23:11:43.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:11:43.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:11:43.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:11:43.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:11:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2024-03-11T23:11:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_28': 2024-03-11T23:11:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:11:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:11:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:11:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 1297ec56607a 2024-03-11T23:11:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.1297ec56607a 2024-03-11T23:11:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:11:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:11:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:11:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:11:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:03:57 2024 2024-03-11T23:11:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:03:57 2024 2024-03-11T23:11:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:03:57 2024 2024-03-11T23:11:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:11:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:11:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:11:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 93b4a8e6-8dee-44ce-b32e-2d1497b47e9c 2024-03-11T23:11:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:11:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:11:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_28 snap 2024-03-11T23:11:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_28 2024-03-11T23:11:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:11:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:11:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_28.export 2024-03-11T23:11:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_28.export 2024-03-11T23:11:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_28.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_28.export 2024-03-11T23:11:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_28.export 2024-03-11T23:11:44.587 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:11:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_28.export 2024-03-11T23:11:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:11:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_28.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_28.export 2024-03-11T23:11:44.878 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_28.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_28.export 2024-03-11T23:11:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:11:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:11:44.909 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_29 2024-03-11T23:11:44.909 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_29 snap 2024-03-11T23:11:44.909 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:44.909 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:44.909 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_29 2024-03-11T23:11:44.909 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:11:44.909 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:11:45.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2024-03-11T23:11:45.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:45.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:45.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_29 2024-03-11T23:11:45.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2024-03-11T23:11:45.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:45.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:45.316 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_29 2024-03-11T23:11:45.316 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:11:45.316 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:11:45.316 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:11:45.316 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:11:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2024-03-11T23:11:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_29 2024-03-11T23:11:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:11:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:11:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:11:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_29 2024-03-11T23:11:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:11:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:11:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:11:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:11:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:11:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:11:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:11:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:11:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:11:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:11:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:11:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2024-03-11T23:11:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:11:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:11:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:11:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:11:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:11:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:11:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:11:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:11:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:11:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:11:46.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:11:46.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:46.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:11:46.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:11:46.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:11:46.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:11:46.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:11:46.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:11:46.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:11:46.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:11:46.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:11:46.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:11:46.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:11:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:11:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:11:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:11:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:11:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:11:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:11:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:11:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:11:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:11:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2024-03-11T23:11:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-03-11T23:11:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:11:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:11:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:11:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_29 2024-03-11T23:11:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:11:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:11:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:11:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:11:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:11:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:11:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:11:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:11:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:11:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:11:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:11:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:11:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:11:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:11:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:11:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:11:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:11:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:11:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:11:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:11:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:11:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:11:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:11:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:11:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:11:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:11:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:11:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:11:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:11:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:11:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:11:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_29 2024-03-11T23:11:46.536 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-03-11T23:11:46.536 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:11:46.536 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:11:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:11:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_29 2024-03-11T23:11:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T23:11:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:11:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:11:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T23:11:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T23:11:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:11:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:11:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:11:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T23:11:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T23:11:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:11:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T23:11:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:11:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T23:11:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T23:11:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:11:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:11:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T23:11:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T23:11:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:11:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:11:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:11:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T23:11:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:11:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:11:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T23:11:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T23:11:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T23:11:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:11:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:11:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:11:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_29 2024-03-11T23:11:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-03-11T23:11:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:11:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:11:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:11:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_29 2024-03-11T23:11:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:3 2024-03-11T23:11:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:11:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:11:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-03-11T23:11:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:3 2024-03-11T23:11:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:11:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:11:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:11:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=3 2024-03-11T23:11:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 3 = cluster1:3 ']' 2024-03-11T23:11:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:11:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=3 2024-03-11T23:11:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:11:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-03-11T23:11:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:3 2024-03-11T23:11:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:11:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:11:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-03-11T23:11:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:3 2024-03-11T23:11:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:11:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:11:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:11:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 3 = cluster1:3 ']' 2024-03-11T23:11:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:11:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:11:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=3 2024-03-11T23:11:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=3 2024-03-11T23:11:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-03-11T23:11:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:11:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:11:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:11:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_29 2024-03-11T23:11:46.751 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:11:46.751 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:11:46.751 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:11:46.751 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:11:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:11:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:11:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:11:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:11:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2024-03-11T23:11:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:11:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_29 2024-03-11T23:11:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:11:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:11:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_29 2024-03-11T23:11:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:11:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_29 2024-03-11T23:11:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_29.status 2024-03-11T23:11:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_29.status 2024-03-11T23:11:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:11:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_29.status 2024-03-11T23:11:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_29.status 2024-03-11T23:11:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_29.status 2024-03-11T23:11:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:11:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_29.status 2024-03-11T23:11:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_29 2024-03-11T23:11:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_29 2024-03-11T23:11:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_29 2024-03-11T23:11:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 142 2024-03-11T23:11:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_29 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_29.status 2024-03-11T23:11:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_29 2024-03-11T23:11:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_29.status 2024-03-11T23:11:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_29 --format xml 2024-03-11T23:11:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:11:47.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=142 2024-03-11T23:11:47.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:11:47.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:11:47.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:11:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_29.status 2024-03-11T23:11:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_29 2024-03-11T23:11:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_29.status 2024-03-11T23:11:47.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-03-11T23:11:47.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:47.626 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:11:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=63 2024-03-11T23:11:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 142 = 63 2024-03-11T23:11:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:11:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:11:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_29.status 2024-03-11T23:11:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_29 2024-03-11T23:11:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_29.status 2024-03-11T23:11:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-03-11T23:11:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:11:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:11:48.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:11:48.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:11:48.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:11:48.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:11:48.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 142 = '' 2024-03-11T23:11:48.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:11:48.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:11:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_29.status 2024-03-11T23:11:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_29 2024-03-11T23:11:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_29.status 2024-03-11T23:11:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-03-11T23:11:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:11:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=142 2024-03-11T23:11:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 142 = 142 2024-03-11T23:11:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:11:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2024-03-11T23:11:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_29 2024-03-11T23:11:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:11:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:11:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:11:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:11:50.145 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2024-03-11T23:11:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_29': 2024-03-11T23:11:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:11:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:11:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:11:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 10b32aa08c14 2024-03-11T23:11:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.10b32aa08c14 2024-03-11T23:11:50.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:11:50.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:11:50.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:11:50.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:11:50.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:04:09 2024 2024-03-11T23:11:50.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:04:09 2024 2024-03-11T23:11:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:04:09 2024 2024-03-11T23:11:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:11:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:11:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:11:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 56839585-8992-435c-80a3-a4d4b2cb652c 2024-03-11T23:11:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:11:50.192 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:11:50.192 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_29 snap 2024-03-11T23:11:50.192 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:50.192 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_29 2024-03-11T23:11:50.192 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:11:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:11:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_29.export 2024-03-11T23:11:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_29.export 2024-03-11T23:11:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_29.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_29.export 2024-03-11T23:11:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_29.export 2024-03-11T23:11:50.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:11:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_29.export 2024-03-11T23:11:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:11:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_29.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_29.export 2024-03-11T23:11:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_29.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_29.export 2024-03-11T23:11:50.683 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:11:50.684 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:11:50.684 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_30 2024-03-11T23:11:50.684 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_30 snap 2024-03-11T23:11:50.684 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:50.684 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:50.684 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_30 2024-03-11T23:11:50.684 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:11:50.684 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:11:51.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2024-03-11T23:11:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_30 2024-03-11T23:11:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2024-03-11T23:11:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_30 2024-03-11T23:11:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:11:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:11:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:11:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:11:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2024-03-11T23:11:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_30 2024-03-11T23:11:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:11:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:11:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:11:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_30 2024-03-11T23:11:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:11:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:11:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:11:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:11:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:11:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:11:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:11:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:11:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:11:52.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:11:52.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:11:52.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2024-03-11T23:11:52.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:11:52.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:11:52.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:11:52.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:11:52.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:11:52.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:11:52.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:11:52.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:11:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:11:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:11:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:11:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:11:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:11:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:11:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:11:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:11:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:11:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:11:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:11:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:11:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:11:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:52.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:11:52.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:11:52.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:11:52.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:11:52.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:11:52.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:11:52.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:11:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:11:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:11:52.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:11:52.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2024-03-11T23:11:52.456 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-03-11T23:11:52.456 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:11:52.456 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:11:52.461 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:11:52.461 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2024-03-11T23:11:52.462 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:11:52.462 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:11:52.462 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:11:52.462 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:11:52.462 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:11:52.462 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:11:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:11:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:11:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:11:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:11:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:11:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:11:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:11:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:11:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:11:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:11:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:11:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:11:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:11:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:11:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:11:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:11:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:11:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:11:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:11:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:11:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:11:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:11:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:11:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:11:52.478 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:11:52.478 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2024-03-11T23:11:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-03-11T23:11:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:11:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:11:52.563 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:11:52.563 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_30 2024-03-11T23:11:52.563 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T23:11:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:11:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:11:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T23:11:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T23:11:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:11:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:11:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:11:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T23:11:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T23:11:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:11:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T23:11:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:11:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T23:11:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T23:11:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:11:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:11:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T23:11:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T23:11:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:11:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:11:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:11:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T23:11:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:11:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:11:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T23:11:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T23:11:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T23:11:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:11:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:11:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:11:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_30 2024-03-11T23:11:52.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-03-11T23:11:52.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:11:52.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:11:52.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:11:52.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_30 2024-03-11T23:11:52.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:3 2024-03-11T23:11:52.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:11:52.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:11:52.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-03-11T23:11:52.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:3 2024-03-11T23:11:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:11:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:11:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:11:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=3 2024-03-11T23:11:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 3 = cluster1:3 ']' 2024-03-11T23:11:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:11:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=3 2024-03-11T23:11:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:11:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-03-11T23:11:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:3 2024-03-11T23:11:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:11:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:11:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-03-11T23:11:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:3 2024-03-11T23:11:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:11:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:11:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:11:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 3 = cluster1:3 ']' 2024-03-11T23:11:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:11:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:11:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=3 2024-03-11T23:11:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=3 2024-03-11T23:11:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-03-11T23:11:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:11:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:11:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:11:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_30 2024-03-11T23:11:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:11:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:11:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:11:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:11:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:11:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:11:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:11:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:11:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2024-03-11T23:11:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:11:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_30 2024-03-11T23:11:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:11:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:11:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_30 2024-03-11T23:11:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:11:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_30 2024-03-11T23:11:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_30.status 2024-03-11T23:11:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_30.status 2024-03-11T23:11:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:11:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_30.status 2024-03-11T23:11:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_30.status 2024-03-11T23:11:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_30.status 2024-03-11T23:11:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:11:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_30.status 2024-03-11T23:11:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_30 2024-03-11T23:11:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_30 2024-03-11T23:11:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_30 2024-03-11T23:11:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 144 2024-03-11T23:11:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_30 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_30.status 2024-03-11T23:11:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_30 2024-03-11T23:11:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_30.status 2024-03-11T23:11:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_30 --format xml 2024-03-11T23:11:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:53.400 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:11:53.402 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=144 2024-03-11T23:11:53.402 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:11:53.402 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:11:53.402 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:11:53.604 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_30.status 2024-03-11T23:11:53.604 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:53.604 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:53.604 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_30 2024-03-11T23:11:53.604 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_30.status 2024-03-11T23:11:53.604 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-03-11T23:11:53.604 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:53.650 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:11:53.652 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=64 2024-03-11T23:11:53.652 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 144 = 64 2024-03-11T23:11:53.652 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:11:53.652 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:11:54.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_30.status 2024-03-11T23:11:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_30 2024-03-11T23:11:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_30.status 2024-03-11T23:11:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-03-11T23:11:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:54.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:11:54.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:11:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:11:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:11:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:11:54.102 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:11:54.102 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 144 = '' 2024-03-11T23:11:54.102 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:11:54.102 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:11:54.903 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_30.status 2024-03-11T23:11:54.903 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:54.903 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:54.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_30 2024-03-11T23:11:54.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_30.status 2024-03-11T23:11:54.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-03-11T23:11:54.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:54.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:11:54.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:11:54.962 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:11:54.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:11:54.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:11:54.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:11:54.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 144 = '' 2024-03-11T23:11:54.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:11:54.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:11:56.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_30.status 2024-03-11T23:11:56.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:56.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:56.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_30 2024-03-11T23:11:56.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_30.status 2024-03-11T23:11:56.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-03-11T23:11:56.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:11:56.618 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:11:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=144 2024-03-11T23:11:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 144 = 144 2024-03-11T23:11:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:11:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2024-03-11T23:11:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_30 2024-03-11T23:11:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:11:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:11:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:11:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:11:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2024-03-11T23:11:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_30': 2024-03-11T23:11:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:11:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:11:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:11:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 11ff31047689 2024-03-11T23:11:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.11ff31047689 2024-03-11T23:11:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:11:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:11:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:11:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:11:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:04:17 2024 2024-03-11T23:11:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:04:17 2024 2024-03-11T23:11:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:04:17 2024 2024-03-11T23:11:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:11:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:11:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:11:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 5b5325b7-4350-47ba-9791-39702d0ec94a 2024-03-11T23:11:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:11:57.670 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:11:57.670 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_30 snap 2024-03-11T23:11:57.670 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:57.670 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_30 2024-03-11T23:11:57.670 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:11:57.670 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:11:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_30.export 2024-03-11T23:11:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_30.export 2024-03-11T23:11:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_30.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_30.export 2024-03-11T23:11:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_30.export 2024-03-11T23:11:57.854 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:11:57.859 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_30.export 2024-03-11T23:11:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:11:58.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_30.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_30.export 2024-03-11T23:11:58.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_30.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_30.export 2024-03-11T23:11:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:11:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:11:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_31 2024-03-11T23:11:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_31 snap 2024-03-11T23:11:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_31 2024-03-11T23:11:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:11:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:11:58.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2024-03-11T23:11:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_31 2024-03-11T23:11:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2024-03-11T23:11:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_31 2024-03-11T23:11:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:11:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:11:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:11:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:11:59.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2024-03-11T23:11:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:11:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_31 2024-03-11T23:11:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:11:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:11:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:11:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_31 2024-03-11T23:11:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:11:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:11:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:11:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:11:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:11:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:11:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:11:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:11:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:11:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:11:59.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:11:59.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2024-03-11T23:11:59.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:11:59.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:11:59.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:11:59.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:11:59.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:11:59.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:11:59.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:11:59.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:11:59.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:11:59.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:11:59.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:11:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:11:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:11:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:11:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:11:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:11:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:11:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:11:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:11:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:11:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:11:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:11:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:11:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:11:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:11:59.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:11:59.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:11:59.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:11:59.383 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:11:59.383 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:11:59.383 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:11:59.384 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2024-03-11T23:11:59.473 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-03-11T23:11:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:11:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:11:59.477 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:11:59.477 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_31 2024-03-11T23:11:59.477 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:11:59.477 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:11:59.477 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:11:59.477 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:11:59.477 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:11:59.477 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:11:59.478 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:11:59.478 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:11:59.478 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:11:59.478 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:11:59.478 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:11:59.478 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:59.478 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:11:59.478 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:11:59.478 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:11:59.478 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:11:59.478 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:11:59.478 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:11:59.478 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:11:59.478 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:11:59.478 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:11:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:11:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:11:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:11:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:11:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:11:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:11:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:11:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:11:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:11:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:11:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:11:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_31 2024-03-11T23:11:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-03-11T23:11:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:11:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:11:59.583 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:11:59.583 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_31 2024-03-11T23:11:59.583 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T23:11:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:11:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:11:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T23:11:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T23:11:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:11:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:11:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:11:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T23:11:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T23:11:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:11:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T23:11:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:11:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T23:11:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T23:11:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:11:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:11:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T23:11:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T23:11:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:11:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:11:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:11:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:11:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T23:11:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:11:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:11:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T23:11:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T23:11:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T23:11:59.602 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:11:59.602 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:11:59.602 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:11:59.602 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_31 2024-03-11T23:11:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:11:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:11:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:11:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:11:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:11:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:11:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:11:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:11:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2024-03-11T23:11:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:11:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_31 2024-03-11T23:11:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:11:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:11:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_31 2024-03-11T23:11:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:11:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_31 2024-03-11T23:11:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_31.status 2024-03-11T23:11:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_31.status 2024-03-11T23:11:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:11:59.697 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_31.status 2024-03-11T23:11:59.697 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_31.status 2024-03-11T23:11:59.697 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_31.status 2024-03-11T23:11:59.697 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:11:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_31.status 2024-03-11T23:11:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_31 2024-03-11T23:11:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:11:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:11:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_31 2024-03-11T23:11:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_31 2024-03-11T23:12:00.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 146 2024-03-11T23:12:00.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_31 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_31.status 2024-03-11T23:12:00.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:12:00.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:00.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_31 2024-03-11T23:12:00.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_31.status 2024-03-11T23:12:00.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_31 --format xml 2024-03-11T23:12:00.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:12:00.414 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:12:00.416 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=146 2024-03-11T23:12:00.416 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:12:00.416 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:12:00.416 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:12:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_31.status 2024-03-11T23:12:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_31 2024-03-11T23:12:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_31.status 2024-03-11T23:12:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-03-11T23:12:00.618 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:12:00.663 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:12:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=65 2024-03-11T23:12:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 146 = 65 2024-03-11T23:12:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:12:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:12:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_31.status 2024-03-11T23:12:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_31 2024-03-11T23:12:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_31.status 2024-03-11T23:12:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-03-11T23:12:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:12:01.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:12:01.113 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:12:01.114 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:12:01.114 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:12:01.114 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:12:01.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:12:01.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 146 = '' 2024-03-11T23:12:01.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:12:01.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:12:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_31.status 2024-03-11T23:12:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_31 2024-03-11T23:12:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_31.status 2024-03-11T23:12:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-03-11T23:12:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:12:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:12:01.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:12:01.975 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:12:01.975 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:12:01.975 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:12:01.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:12:01.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 146 = '' 2024-03-11T23:12:01.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:12:01.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:12:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_31.status 2024-03-11T23:12:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_31 2024-03-11T23:12:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_31.status 2024-03-11T23:12:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-03-11T23:12:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:12:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:12:03.631 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=146 2024-03-11T23:12:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 146 = 146 2024-03-11T23:12:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:12:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2024-03-11T23:12:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_31 2024-03-11T23:12:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:12:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:12:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:12:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:12:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2024-03-11T23:12:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_31': 2024-03-11T23:12:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:12:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:12:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:12:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 12608d86915c 2024-03-11T23:12:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.12608d86915c 2024-03-11T23:12:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:12:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:12:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:12:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:12:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:04:26 2024 2024-03-11T23:12:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:04:26 2024 2024-03-11T23:12:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:04:26 2024 2024-03-11T23:12:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:12:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:12:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:12:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 0fbab2fa-8751-419f-a846-c51d8786c5c5 2024-03-11T23:12:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:12:04.681 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:12:04.681 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_31 snap 2024-03-11T23:12:04.681 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:04.681 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_31 2024-03-11T23:12:04.681 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:12:04.681 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:12:04.681 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_31.export 2024-03-11T23:12:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_31.export 2024-03-11T23:12:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_31.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_31.export 2024-03-11T23:12:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_31.export 2024-03-11T23:12:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:12:04.877 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_31.export 2024-03-11T23:12:05.117 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:12:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_31.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_31.export 2024-03-11T23:12:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_31.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_31.export 2024-03-11T23:12:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:12:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:12:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_32 2024-03-11T23:12:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_32 snap 2024-03-11T23:12:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:12:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_32 2024-03-11T23:12:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:12:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:12:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2024-03-11T23:12:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_32 2024-03-11T23:12:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2024-03-11T23:12:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_32 2024-03-11T23:12:05.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:12:05.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:12:05.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:12:05.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:12:06.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2024-03-11T23:12:06.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:06.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:06.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_32 2024-03-11T23:12:06.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:12:06.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:12:06.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:12:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_32 2024-03-11T23:12:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:12:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:12:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:12:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:12:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:12:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:12:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:12:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:12:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:12:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:12:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:12:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2024-03-11T23:12:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:12:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:12:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:12:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:12:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:12:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:12:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:12:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:12:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:12:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:12:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:12:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:12:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:12:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:12:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:12:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:12:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:12:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:12:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:12:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:12:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:12:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:12:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:12:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:12:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:12:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:12:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:12:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:12:06.394 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:12:06.394 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:12:06.394 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:12:06.394 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2024-03-11T23:12:06.483 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-03-11T23:12:06.483 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:12:06.483 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:12:06.487 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:12:06.487 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_32 2024-03-11T23:12:06.487 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:12:06.487 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:12:06.487 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:12:06.487 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:12:06.487 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:12:06.487 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:12:06.487 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:12:06.487 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:12:06.487 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:12:06.487 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:12:06.487 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:12:06.488 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:06.488 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:12:06.488 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:12:06.488 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:12:06.488 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:12:06.488 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:12:06.488 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:12:06.488 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:12:06.488 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:12:06.488 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:12:06.488 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:12:06.488 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:06.488 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:12:06.488 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:12:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:12:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:12:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:12:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:12:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:12:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:12:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:12:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:12:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_32 2024-03-11T23:12:06.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:12:06.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:12:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:12:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:12:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:12:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:12:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:12:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:12:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2024-03-11T23:12:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:12:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:12:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_32 2024-03-11T23:12:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:12:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:12:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_32 2024-03-11T23:12:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:12:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:12:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_32 2024-03-11T23:12:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_32.status 2024-03-11T23:12:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_32.status 2024-03-11T23:12:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:12:06.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_32.status 2024-03-11T23:12:06.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_32.status 2024-03-11T23:12:06.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_32.status 2024-03-11T23:12:06.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:12:06.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_32.status 2024-03-11T23:12:06.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_32 2024-03-11T23:12:06.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:12:06.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:06.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_32 2024-03-11T23:12:06.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_32 2024-03-11T23:12:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 148 2024-03-11T23:12:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_32 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_32.status 2024-03-11T23:12:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:12:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_32 2024-03-11T23:12:07.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_32.status 2024-03-11T23:12:07.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_32 --format xml 2024-03-11T23:12:07.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:12:07.430 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:12:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=148 2024-03-11T23:12:07.432 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:12:07.432 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:12:07.432 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:12:07.633 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_32.status 2024-03-11T23:12:07.633 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:07.633 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:07.633 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_32 2024-03-11T23:12:07.633 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_32.status 2024-03-11T23:12:07.633 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-03-11T23:12:07.633 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:12:07.680 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:12:07.682 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=66 2024-03-11T23:12:07.682 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 148 = 66 2024-03-11T23:12:07.682 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:12:07.682 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:12:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_32.status 2024-03-11T23:12:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_32 2024-03-11T23:12:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_32.status 2024-03-11T23:12:08.084 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-03-11T23:12:08.084 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:12:08.131 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:12:08.131 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:12:08.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:12:08.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:12:08.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:12:08.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:12:08.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 148 = '' 2024-03-11T23:12:08.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:12:08.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:12:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_32.status 2024-03-11T23:12:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_32 2024-03-11T23:12:08.935 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_32.status 2024-03-11T23:12:08.935 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-03-11T23:12:08.935 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:12:08.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:12:08.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:12:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:12:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:12:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:12:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:12:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 148 = '' 2024-03-11T23:12:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:12:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:12:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_32.status 2024-03-11T23:12:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_32 2024-03-11T23:12:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_32.status 2024-03-11T23:12:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-03-11T23:12:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:12:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:12:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=148 2024-03-11T23:12:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 148 = 148 2024-03-11T23:12:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:12:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2024-03-11T23:12:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_32 2024-03-11T23:12:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:12:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:12:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:12:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:12:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2024-03-11T23:12:11.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_32': 2024-03-11T23:12:11.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:12:11.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:12:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:12:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 12973a441cff 2024-03-11T23:12:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.12973a441cff 2024-03-11T23:12:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:12:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:12:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:12:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:12:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:04:33 2024 2024-03-11T23:12:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:04:33 2024 2024-03-11T23:12:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:04:33 2024 2024-03-11T23:12:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:12:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:12:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:12:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: adeee68d-39bd-4c47-9b90-d83d5ac98497 2024-03-11T23:12:11.702 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:12:11.704 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:12:11.704 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_32 snap 2024-03-11T23:12:11.704 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_32 2024-03-11T23:12:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:12:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:12:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_32.export 2024-03-11T23:12:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_32.export 2024-03-11T23:12:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_32.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_32.export 2024-03-11T23:12:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_32.export 2024-03-11T23:12:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:12:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_32.export 2024-03-11T23:12:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:12:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_32.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_32.export 2024-03-11T23:12:12.169 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_32.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_32.export 2024-03-11T23:12:12.199 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:12:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:12:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_33 2024-03-11T23:12:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_33 snap 2024-03-11T23:12:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:12:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_33 2024-03-11T23:12:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:12:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:12:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2024-03-11T23:12:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_33 2024-03-11T23:12:12.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2024-03-11T23:12:12.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:12.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:12.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_33 2024-03-11T23:12:12.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:12:12.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:12:12.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:12:12.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:12:13.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2024-03-11T23:12:13.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_33 2024-03-11T23:12:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:12:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:12:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:12:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_33 2024-03-11T23:12:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:12:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:12:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:12:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:12:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:12:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:12:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:12:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:12:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:12:13.385 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:12:13.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:12:13.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2024-03-11T23:12:13.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:12:13.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:12:13.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:12:13.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:12:13.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:12:13.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:12:13.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:12:13.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:12:13.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:12:13.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:12:13.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:12:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:12:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:12:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:12:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:12:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:12:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:12:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:12:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:12:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:12:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:12:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:12:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:12:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:12:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:12:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:12:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:12:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:12:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:12:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:12:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:12:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2024-03-11T23:12:13.492 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-03-11T23:12:13.492 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:12:13.492 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:12:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:12:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2024-03-11T23:12:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:12:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:12:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:12:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:12:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:12:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:12:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:12:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:12:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:12:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:12:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:12:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:12:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:12:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:12:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:12:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:12:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:12:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:12:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:12:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:12:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:12:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:12:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:12:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:12:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:12:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:12:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:12:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:12:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:12:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:12:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:12:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2024-03-11T23:12:13.597 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-03-11T23:12:13.597 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:12:13.597 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:12:13.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:12:13.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_33 2024-03-11T23:12:13.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T23:12:13.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:12:13.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:12:13.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T23:12:13.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T23:12:13.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:12:13.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:12:13.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:12:13.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T23:12:13.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T23:12:13.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:12:13.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:13.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T23:12:13.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:12:13.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T23:12:13.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T23:12:13.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:12:13.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:12:13.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T23:12:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T23:12:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:12:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:12:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:12:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T23:12:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:12:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:12:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T23:12:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T23:12:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T23:12:13.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:12:13.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:12:13.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:12:13.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_33 2024-03-11T23:12:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-03-11T23:12:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:12:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:12:13.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:12:13.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_33 2024-03-11T23:12:13.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:3 2024-03-11T23:12:13.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:12:13.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:12:13.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-03-11T23:12:13.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:3 2024-03-11T23:12:13.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:12:13.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:12:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:12:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=3 2024-03-11T23:12:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 3 = cluster1:3 ']' 2024-03-11T23:12:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:12:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=3 2024-03-11T23:12:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:12:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-03-11T23:12:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:3 2024-03-11T23:12:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:12:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:12:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-03-11T23:12:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:3 2024-03-11T23:12:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:12:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:12:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:12:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 3 = cluster1:3 ']' 2024-03-11T23:12:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:12:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:12:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=3 2024-03-11T23:12:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=3 2024-03-11T23:12:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-03-11T23:12:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:12:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:12:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:12:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_33 2024-03-11T23:12:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:12:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:12:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:12:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:12:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:12:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:12:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:12:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:12:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2024-03-11T23:12:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:12:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:12:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_33 2024-03-11T23:12:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:12:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:12:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_33 2024-03-11T23:12:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:12:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:12:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_33 2024-03-11T23:12:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_33.status 2024-03-11T23:12:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_33.status 2024-03-11T23:12:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:12:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_33.status 2024-03-11T23:12:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_33.status 2024-03-11T23:12:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_33.status 2024-03-11T23:12:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:12:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_33.status 2024-03-11T23:12:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_33 2024-03-11T23:12:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:12:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_33 2024-03-11T23:12:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_33 2024-03-11T23:12:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 150 2024-03-11T23:12:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_33 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_33.status 2024-03-11T23:12:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:12:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_33 2024-03-11T23:12:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_33.status 2024-03-11T23:12:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_33 --format xml 2024-03-11T23:12:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:12:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:12:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=150 2024-03-11T23:12:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:12:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:12:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:12:14.646 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_33.status 2024-03-11T23:12:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_33 2024-03-11T23:12:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_33.status 2024-03-11T23:12:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-03-11T23:12:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:12:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:12:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=67 2024-03-11T23:12:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 150 = 67 2024-03-11T23:12:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:12:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:12:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_33.status 2024-03-11T23:12:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_33 2024-03-11T23:12:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_33.status 2024-03-11T23:12:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-03-11T23:12:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:12:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:12:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:12:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:12:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:12:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:12:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:12:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 150 = '' 2024-03-11T23:12:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:12:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:12:15.833 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-03-11T23:12:15.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-03-11T23:12:15.834 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-03-11T23:12:15.834 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-03-11T23:12:15.834 INFO:teuthology.orchestra.run:waiting for 600 2024-03-11T23:12:15.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi132.stderr:2024-03-11T23:12:15.831+0000 7f4a97a40640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 53631) UID: 1000 2024-03-11T23:12:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_33.status 2024-03-11T23:12:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_33 2024-03-11T23:12:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_33.status 2024-03-11T23:12:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-03-11T23:12:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:12:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:12:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:12:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:12:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:12:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:12:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:12:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 150 = '' 2024-03-11T23:12:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:12:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:12:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_33.status 2024-03-11T23:12:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_33 2024-03-11T23:12:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_33.status 2024-03-11T23:12:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-03-11T23:12:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:12:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:12:17.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=150 2024-03-11T23:12:17.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 150 = 150 2024-03-11T23:12:17.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:12:17.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2024-03-11T23:12:17.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:17.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:17.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_33 2024-03-11T23:12:17.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:12:17.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:12:17.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:12:17.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:12:18.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2024-03-11T23:12:18.704 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_33': 2024-03-11T23:12:18.704 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:12:18.704 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:12:18.704 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:12:18.704 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 10b3472e6ebf 2024-03-11T23:12:18.704 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.10b3472e6ebf 2024-03-11T23:12:18.704 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:12:18.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:12:18.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:12:18.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:12:18.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:04:43 2024 2024-03-11T23:12:18.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:04:43 2024 2024-03-11T23:12:18.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:04:43 2024 2024-03-11T23:12:18.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:12:18.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:12:18.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:12:18.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 94a5319a-2e74-4f65-a321-3f962cd4ebe5 2024-03-11T23:12:18.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:12:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:12:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_33 snap 2024-03-11T23:12:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_33 2024-03-11T23:12:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:12:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:12:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_33.export 2024-03-11T23:12:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_33.export 2024-03-11T23:12:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_33.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_33.export 2024-03-11T23:12:18.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_33.export 2024-03-11T23:12:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:12:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_33.export 2024-03-11T23:12:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:12:19.147 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_33.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_33.export 2024-03-11T23:12:19.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_33.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_33.export 2024-03-11T23:12:19.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:12:19.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:12:19.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_34 2024-03-11T23:12:19.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_34 snap 2024-03-11T23:12:19.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:12:19.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:19.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_34 2024-03-11T23:12:19.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:12:19.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:12:19.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2024-03-11T23:12:19.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_34 2024-03-11T23:12:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2024-03-11T23:12:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_34 2024-03-11T23:12:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:12:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:12:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:12:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:12:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2024-03-11T23:12:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_34 2024-03-11T23:12:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:12:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:12:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:12:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_34 2024-03-11T23:12:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:12:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:12:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:12:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:12:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:12:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:12:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:12:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:12:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:12:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:12:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:12:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2024-03-11T23:12:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:12:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:12:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:12:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:12:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:12:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:12:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:12:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:12:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:12:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:12:20.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:12:20.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:20.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:12:20.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:12:20.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:12:20.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:12:20.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:12:20.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:12:20.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:12:20.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:12:20.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:12:20.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:12:20.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:20.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:12:20.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:12:20.406 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:12:20.406 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:12:20.406 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:12:20.406 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:12:20.406 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:12:20.425 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:12:20.425 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:12:20.425 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:12:20.425 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2024-03-11T23:12:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-03-11T23:12:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:12:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2024-03-11T23:12:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:12:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:12:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:12:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:12:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:12:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:12:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:12:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:12:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:12:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:12:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:12:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:12:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:12:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:12:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:12:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:12:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:12:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:12:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:12:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:12:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:12:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:12:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:12:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:12:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:12:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:12:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:12:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:12:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:12:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:12:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:12:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2024-03-11T23:12:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:12:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:12:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:12:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:12:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:12:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:12:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:12:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:12:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_34 2024-03-11T23:12:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:12:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:12:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_34 2024-03-11T23:12:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:12:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:12:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_34 2024-03-11T23:12:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:12:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:12:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_34 2024-03-11T23:12:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_34.status 2024-03-11T23:12:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_34.status 2024-03-11T23:12:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:12:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_34.status 2024-03-11T23:12:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_34.status 2024-03-11T23:12:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_34.status 2024-03-11T23:12:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:12:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_34.status 2024-03-11T23:12:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_34 2024-03-11T23:12:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:12:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_34 2024-03-11T23:12:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_34 2024-03-11T23:12:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 152 2024-03-11T23:12:21.416 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_34 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_34.status 2024-03-11T23:12:21.416 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:12:21.416 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:21.416 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_34 2024-03-11T23:12:21.416 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_34.status 2024-03-11T23:12:21.416 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_34 --format xml 2024-03-11T23:12:21.416 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:12:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:12:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=152 2024-03-11T23:12:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:12:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:12:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:12:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_34.status 2024-03-11T23:12:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_34 2024-03-11T23:12:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_34.status 2024-03-11T23:12:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-03-11T23:12:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:12:21.713 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:12:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=68 2024-03-11T23:12:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 152 = 68 2024-03-11T23:12:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:12:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:12:21.936 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-03-11T23:12:21.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-03-11T23:12:21.936 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-03-11T23:12:21.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-03-11T23:12:21.939 INFO:tasks.rbd_mirror_thrash:waiting for 93 secs before thrashing 2024-03-11T23:12:22.116 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_34.status 2024-03-11T23:12:22.116 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:22.116 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:22.116 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_34 2024-03-11T23:12:22.116 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_34.status 2024-03-11T23:12:22.116 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-03-11T23:12:22.116 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:12:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:12:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:12:22.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:12:22.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:12:22.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:12:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:12:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 152 = '' 2024-03-11T23:12:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:12:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:12:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_34.status 2024-03-11T23:12:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_34 2024-03-11T23:12:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_34.status 2024-03-11T23:12:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-03-11T23:12:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:12:23.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:12:23.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:12:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:12:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:12:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:12:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:12:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 152 = '' 2024-03-11T23:12:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:12:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:12:24.615 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_34.status 2024-03-11T23:12:24.615 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:24.615 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:24.615 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_34 2024-03-11T23:12:24.615 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_34.status 2024-03-11T23:12:24.615 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-03-11T23:12:24.615 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:12:24.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:12:24.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=152 2024-03-11T23:12:24.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 152 = 152 2024-03-11T23:12:24.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:12:24.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2024-03-11T23:12:24.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:24.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:24.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_34 2024-03-11T23:12:24.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:12:24.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:12:24.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:12:24.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:12:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2024-03-11T23:12:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_34': 2024-03-11T23:12:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:12:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:12:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:12:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 11ff1255ffd5 2024-03-11T23:12:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.11ff1255ffd5 2024-03-11T23:12:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:12:25.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:12:25.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:12:25.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:12:25.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:04:48 2024 2024-03-11T23:12:25.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:04:48 2024 2024-03-11T23:12:25.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:04:48 2024 2024-03-11T23:12:25.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:12:25.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:12:25.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:12:25.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: fa15ec82-36d2-421d-b90c-12034ab39d68 2024-03-11T23:12:25.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:12:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:12:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_34 snap 2024-03-11T23:12:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_34 2024-03-11T23:12:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:12:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:12:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_34.export 2024-03-11T23:12:25.719 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_34.export 2024-03-11T23:12:25.719 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_34.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_34.export 2024-03-11T23:12:25.719 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_34.export 2024-03-11T23:12:25.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:12:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_34.export 2024-03-11T23:12:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:12:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_34.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_34.export 2024-03-11T23:12:26.214 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_34.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_34.export 2024-03-11T23:12:26.244 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:12:26.244 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:12:26.244 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_35 2024-03-11T23:12:26.244 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_35 snap 2024-03-11T23:12:26.244 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:12:26.244 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:26.244 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_35 2024-03-11T23:12:26.244 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:12:26.244 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:12:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2024-03-11T23:12:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_35 2024-03-11T23:12:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2024-03-11T23:12:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_35 2024-03-11T23:12:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:12:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:12:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:12:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:12:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2024-03-11T23:12:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_35 2024-03-11T23:12:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:12:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:12:27.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:12:27.957 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_35 2024-03-11T23:12:27.957 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:12:27.957 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:12:27.957 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:12:27.957 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:12:27.957 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:12:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:12:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:12:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:12:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:12:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:12:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:12:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2024-03-11T23:12:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:12:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:12:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:12:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:12:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:12:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:12:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:12:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:12:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:12:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:12:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:12:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:12:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:12:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:12:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:12:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:12:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:12:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:12:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:12:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:12:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:12:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:12:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:12:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:12:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:12:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:12:27.962 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:12:27.962 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:12:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:12:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:12:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:12:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2024-03-11T23:12:28.073 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:12:28.073 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:12:28.073 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:12:28.073 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:12:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:12:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:12:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:12:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:12:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_35 2024-03-11T23:12:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:12:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:12:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_35 2024-03-11T23:12:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:12:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:12:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_35 2024-03-11T23:12:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:12:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:12:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_35 2024-03-11T23:12:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_35.status 2024-03-11T23:12:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_35.status 2024-03-11T23:12:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:12:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_35.status 2024-03-11T23:12:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_35.status 2024-03-11T23:12:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_35.status 2024-03-11T23:12:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:12:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_35.status 2024-03-11T23:12:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_35 2024-03-11T23:12:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:12:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_35 2024-03-11T23:12:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_35 2024-03-11T23:12:28.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 154 2024-03-11T23:12:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_35 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_35.status 2024-03-11T23:12:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:12:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_35 2024-03-11T23:12:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_35.status 2024-03-11T23:12:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_35 --format xml 2024-03-11T23:12:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:12:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:12:29.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=154 2024-03-11T23:12:29.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:12:29.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:12:29.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:12:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_35.status 2024-03-11T23:12:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_35 2024-03-11T23:12:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_35.status 2024-03-11T23:12:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-03-11T23:12:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:12:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:12:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:12:29.272 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:12:29.272 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:12:29.272 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:12:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:12:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 154 = '' 2024-03-11T23:12:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:12:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:12:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_35.status 2024-03-11T23:12:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_35 2024-03-11T23:12:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_35.status 2024-03-11T23:12:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-03-11T23:12:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:12:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:12:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:12:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:12:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:12:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:12:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:12:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 154 = '' 2024-03-11T23:12:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:12:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:12:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_35.status 2024-03-11T23:12:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_35 2024-03-11T23:12:30.530 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_35.status 2024-03-11T23:12:30.530 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-03-11T23:12:30.530 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:12:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:12:30.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=154 2024-03-11T23:12:30.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 154 = 154 2024-03-11T23:12:30.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:12:30.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2024-03-11T23:12:30.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:30.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:30.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_35 2024-03-11T23:12:30.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:12:30.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:12:30.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:12:30.578 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:12:31.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2024-03-11T23:12:31.624 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_35': 2024-03-11T23:12:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:12:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:12:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:12:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 1260cced7c8d 2024-03-11T23:12:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.1260cced7c8d 2024-03-11T23:12:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:12:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:12:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:12:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:12:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:04:59 2024 2024-03-11T23:12:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:04:59 2024 2024-03-11T23:12:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:04:59 2024 2024-03-11T23:12:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:12:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:12:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:12:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: ddb7a161-de34-428e-ae0c-47939be794d1 2024-03-11T23:12:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:12:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:12:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_35 snap 2024-03-11T23:12:31.629 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:31.629 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_35 2024-03-11T23:12:31.629 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:12:31.629 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:12:31.629 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_35.export 2024-03-11T23:12:31.629 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_35.export 2024-03-11T23:12:31.629 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_35.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_35.export 2024-03-11T23:12:31.629 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_35.export 2024-03-11T23:12:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:12:31.834 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_35.export 2024-03-11T23:12:32.059 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:12:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_35.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_35.export 2024-03-11T23:12:32.121 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_35.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_35.export 2024-03-11T23:12:32.151 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:12:32.151 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:12:32.151 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_36 2024-03-11T23:12:32.151 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_36 snap 2024-03-11T23:12:32.151 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:12:32.151 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_36 2024-03-11T23:12:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:12:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:12:32.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2024-03-11T23:12:32.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:32.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:32.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_36 2024-03-11T23:12:32.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2024-03-11T23:12:32.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:32.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:32.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_36 2024-03-11T23:12:32.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:12:32.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:12:32.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:12:32.977 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:12:33.926 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-03-11T23:12:33.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-03-11T23:12:33.927 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-03-11T23:12:33.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr:2024-03-11T23:12:33.924+0000 7f311c4b7640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 57599) UID: 1000 2024-03-11T23:12:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2024-03-11T23:12:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_36 2024-03-11T23:12:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:12:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:12:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:12:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_36 2024-03-11T23:12:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:12:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:12:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:12:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:12:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:12:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:12:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:12:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:12:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:12:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:12:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:12:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2024-03-11T23:12:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:12:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:12:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:12:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:12:33.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:12:33.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:12:33.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:12:33.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:12:33.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:12:33.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:12:33.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:12:33.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:33.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:12:33.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:12:33.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:12:33.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:12:33.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:12:33.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:12:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:12:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:12:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:12:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:12:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:12:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:12:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:12:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:12:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:12:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:12:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:12:34.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:12:34.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:12:34.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:12:34.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2024-03-11T23:12:34.082 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-03-11T23:12:34.082 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:12:34.082 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:12:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:12:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2024-03-11T23:12:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:12:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:12:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:12:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:12:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:12:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:12:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:12:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:12:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:12:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:12:34.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:12:34.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:34.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:12:34.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:12:34.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:12:34.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:12:34.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:12:34.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:12:34.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:12:34.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:12:34.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:12:34.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:12:34.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:34.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:12:34.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:12:34.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:12:34.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:12:34.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:12:34.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:12:34.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:12:34.103 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:12:34.103 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:12:34.103 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:12:34.103 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2024-03-11T23:12:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-03-11T23:12:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:12:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:12:34.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:12:34.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_36 2024-03-11T23:12:34.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T23:12:34.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:12:34.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:12:34.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T23:12:34.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T23:12:34.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:12:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:12:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:12:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T23:12:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T23:12:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:12:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T23:12:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:12:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T23:12:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T23:12:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:12:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:12:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T23:12:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T23:12:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:12:34.192 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:12:34.192 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:34.192 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:12:34.192 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T23:12:34.192 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:12:34.192 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:12:34.192 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T23:12:34.192 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T23:12:34.192 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T23:12:34.207 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:12:34.207 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:12:34.207 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:12:34.207 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_36 2024-03-11T23:12:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-03-11T23:12:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:12:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:12:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:12:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_36 2024-03-11T23:12:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:3 2024-03-11T23:12:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:12:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:12:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-03-11T23:12:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:3 2024-03-11T23:12:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:12:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:12:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:12:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=3 2024-03-11T23:12:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 3 = cluster1:3 ']' 2024-03-11T23:12:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:12:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=3 2024-03-11T23:12:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:12:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-03-11T23:12:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:3 2024-03-11T23:12:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:12:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:12:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-03-11T23:12:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:3 2024-03-11T23:12:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:12:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:12:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:12:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 3 = cluster1:3 ']' 2024-03-11T23:12:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:12:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:12:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=3 2024-03-11T23:12:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=3 2024-03-11T23:12:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-03-11T23:12:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:12:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:12:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:12:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_36 2024-03-11T23:12:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-03-11T23:12:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:12:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:12:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:12:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 1 2024-03-11T23:12:34.927 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-03-11T23:12:34.927 INFO:teuthology.orchestra.run:waiting for 600 2024-03-11T23:12:34.928 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-03-11T23:12:34.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-03-11T23:12:34.928 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-03-11T23:12:34.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-03-11T23:12:34.931 INFO:tasks.rbd_mirror_thrash:waiting for 84 secs before thrashing 2024-03-11T23:12:35.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:12:35.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:12:35.395 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:12:35.395 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2024-03-11T23:12:35.395 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:12:35.395 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:12:35.395 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:12:35.395 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:12:35.395 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:12:35.395 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:12:35.396 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:12:35.396 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:12:35.396 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:12:35.396 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:12:35.396 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:12:35.396 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:35.396 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:12:35.396 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:12:35.396 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:12:35.396 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:12:35.396 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:12:35.396 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:12:35.396 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:12:35.396 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:12:35.396 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:12:35.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:12:35.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:35.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:12:35.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:12:35.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:12:35.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:12:35.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:12:35.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:12:35.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:12:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:12:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:12:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:12:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2024-03-11T23:12:35.492 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-03-11T23:12:35.492 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:12:35.492 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:12:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:12:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2024-03-11T23:12:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:12:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:12:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:12:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:12:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:12:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:12:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:12:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:12:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:12:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:12:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:12:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:12:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:12:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:12:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:12:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:12:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:12:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:12:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:12:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:12:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:12:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:12:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:12:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:12:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:12:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:12:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:12:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:12:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:12:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:12:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:12:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2024-03-11T23:12:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-03-11T23:12:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:12:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:12:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:12:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_36 2024-03-11T23:12:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T23:12:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:12:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:12:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T23:12:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T23:12:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:12:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:12:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:12:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T23:12:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T23:12:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:12:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T23:12:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:12:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T23:12:35.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T23:12:35.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:12:35.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:12:35.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T23:12:35.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T23:12:35.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:12:35.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:12:35.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:35.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:12:35.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T23:12:35.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:12:35.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:12:35.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T23:12:35.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T23:12:35.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T23:12:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:12:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:12:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:12:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_36 2024-03-11T23:12:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-03-11T23:12:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:12:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:12:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:12:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_36 2024-03-11T23:12:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:3 2024-03-11T23:12:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:12:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:12:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-03-11T23:12:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:3 2024-03-11T23:12:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:12:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:12:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:12:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=3 2024-03-11T23:12:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 3 = cluster1:3 ']' 2024-03-11T23:12:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:12:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=3 2024-03-11T23:12:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:12:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-03-11T23:12:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:3 2024-03-11T23:12:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:12:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:12:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-03-11T23:12:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:3 2024-03-11T23:12:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:12:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:12:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:12:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 3 = cluster1:3 ']' 2024-03-11T23:12:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:12:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:12:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=3 2024-03-11T23:12:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=3 2024-03-11T23:12:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-03-11T23:12:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:12:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:12:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:12:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_36 2024-03-11T23:12:35.801 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-03-11T23:12:35.801 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:12:35.801 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:12:35.802 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:12:35.802 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 2 2024-03-11T23:12:37.804 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:12:37.804 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:12:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:12:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2024-03-11T23:12:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:12:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:12:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:12:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:12:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:12:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:12:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:12:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:12:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:12:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:12:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:12:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:12:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:12:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:12:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:12:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:12:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:12:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:12:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:12:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:12:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:12:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:12:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:12:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:12:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:12:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:12:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:12:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:12:37.827 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:12:37.827 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:12:37.827 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:12:37.827 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2024-03-11T23:12:37.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-03-11T23:12:37.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:12:37.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:12:37.916 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:12:37.916 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2024-03-11T23:12:37.916 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:12:37.916 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:12:37.916 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:12:37.916 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:12:37.916 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:12:37.916 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:12:37.916 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:12:37.916 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:12:37.916 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:12:37.916 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:12:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:12:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:12:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:12:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:12:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:12:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:12:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:12:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:12:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:12:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:12:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:12:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:12:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:12:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:12:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:12:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:12:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:12:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:12:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:12:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:12:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:12:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2024-03-11T23:12:38.018 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-03-11T23:12:38.018 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:12:38.018 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:12:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:12:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_36 2024-03-11T23:12:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T23:12:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:12:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:12:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T23:12:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T23:12:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:12:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:12:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:12:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T23:12:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T23:12:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:12:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T23:12:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:12:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T23:12:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T23:12:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:12:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:12:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T23:12:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T23:12:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:12:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:12:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:12:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T23:12:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:12:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:12:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T23:12:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T23:12:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T23:12:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:12:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:12:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:12:38.040 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_36 2024-03-11T23:12:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-03-11T23:12:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:12:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:12:38.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:12:38.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_36 2024-03-11T23:12:38.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:3 2024-03-11T23:12:38.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:12:38.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:12:38.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-03-11T23:12:38.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:3 2024-03-11T23:12:38.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:12:38.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:12:38.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:12:38.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=3 2024-03-11T23:12:38.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 3 = cluster1:3 ']' 2024-03-11T23:12:38.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:12:38.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=3 2024-03-11T23:12:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:12:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-03-11T23:12:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:3 2024-03-11T23:12:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:12:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:12:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-03-11T23:12:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:3 2024-03-11T23:12:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:12:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:12:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:12:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 3 = cluster1:3 ']' 2024-03-11T23:12:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:12:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:12:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=3 2024-03-11T23:12:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=3 2024-03-11T23:12:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-03-11T23:12:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:12:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:12:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:12:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_36 2024-03-11T23:12:38.224 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-03-11T23:12:38.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:12:38.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:12:38.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:12:38.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 4 2024-03-11T23:12:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:12:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:12:42.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:12:42.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2024-03-11T23:12:42.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:12:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:12:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:12:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:12:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:12:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:12:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:12:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:12:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:12:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:12:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:12:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:12:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:12:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:12:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:12:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:12:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:12:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:12:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:12:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:12:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:12:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:12:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:12:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:12:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:12:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:12:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:12:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:12:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:12:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:12:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:12:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2024-03-11T23:12:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-03-11T23:12:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:12:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:12:42.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:12:42.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2024-03-11T23:12:42.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:12:42.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:12:42.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:12:42.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:12:42.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:12:42.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:12:42.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:12:42.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:12:42.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:12:42.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:12:42.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:12:42.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:42.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:12:42.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:12:42.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:12:42.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:12:42.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:12:42.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:12:42.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:12:42.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:12:42.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:12:42.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:12:42.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:42.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:12:42.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:12:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:12:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:12:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:12:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:12:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:12:42.359 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:12:42.359 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:12:42.359 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:12:42.359 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2024-03-11T23:12:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-03-11T23:12:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:12:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:12:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:12:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_36 2024-03-11T23:12:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T23:12:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:12:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:12:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T23:12:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T23:12:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:12:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:12:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:12:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T23:12:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T23:12:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:12:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T23:12:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:12:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T23:12:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T23:12:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:12:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:12:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T23:12:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T23:12:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:12:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:12:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:12:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T23:12:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:12:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:12:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T23:12:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T23:12:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T23:12:42.461 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:12:42.462 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:12:42.462 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:12:42.462 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_36 2024-03-11T23:12:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-03-11T23:12:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:12:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:12:42.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:12:42.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_36 2024-03-11T23:12:42.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:3 2024-03-11T23:12:42.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:12:42.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:12:42.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-03-11T23:12:42.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:3 2024-03-11T23:12:42.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:12:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:12:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:12:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=3 2024-03-11T23:12:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 3 = cluster1:3 ']' 2024-03-11T23:12:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:12:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=3 2024-03-11T23:12:42.551 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:12:42.551 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-03-11T23:12:42.551 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:3 2024-03-11T23:12:42.551 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:12:42.551 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:12:42.551 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-03-11T23:12:42.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:3 2024-03-11T23:12:42.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:12:42.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:12:42.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:42.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:12:42.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 3 = cluster1:3 ']' 2024-03-11T23:12:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:12:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:12:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=3 2024-03-11T23:12:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=3 2024-03-11T23:12:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-03-11T23:12:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:12:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:12:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:12:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_36 2024-03-11T23:12:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-03-11T23:12:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:12:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:12:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:12:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 8 2024-03-11T23:12:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:12:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:12:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:12:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2024-03-11T23:12:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:12:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:12:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:12:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:12:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:12:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:12:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:12:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:12:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:12:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:12:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:12:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:12:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:12:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:12:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:12:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:12:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:12:50.655 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:12:50.655 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:12:50.655 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:12:50.655 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:12:50.655 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:50.655 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:12:50.655 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:12:50.655 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:12:50.655 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:12:50.655 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:12:50.655 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:12:50.655 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:12:50.673 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:12:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:12:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:12:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2024-03-11T23:12:50.770 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:12:50.770 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:12:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:12:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:12:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:12:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:12:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:12:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:12:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_36 2024-03-11T23:12:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:12:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:12:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_36 2024-03-11T23:12:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:12:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:12:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_36 2024-03-11T23:12:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:12:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:12:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_36 2024-03-11T23:12:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_36.status 2024-03-11T23:12:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_36.status 2024-03-11T23:12:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:12:50.775 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_36.status 2024-03-11T23:12:50.775 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_36.status 2024-03-11T23:12:50.775 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_36.status 2024-03-11T23:12:50.775 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:12:50.776 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_36.status 2024-03-11T23:12:50.776 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_36 2024-03-11T23:12:50.776 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:12:50.776 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:50.776 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_36 2024-03-11T23:12:50.776 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_36 2024-03-11T23:12:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 156 2024-03-11T23:12:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_36 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_36.status 2024-03-11T23:12:51.006 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:12:51.006 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:51.006 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_36 2024-03-11T23:12:51.006 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_36.status 2024-03-11T23:12:51.006 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_36 --format xml 2024-03-11T23:12:51.006 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:12:51.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:12:51.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=156 2024-03-11T23:12:51.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:12:51.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:12:51.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:12:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_36.status 2024-03-11T23:12:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_36 2024-03-11T23:12:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_36.status 2024-03-11T23:12:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-03-11T23:12:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:12:51.294 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:12:51.294 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:12:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:12:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:12:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:12:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:12:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 156 = '' 2024-03-11T23:12:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:12:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:12:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_36.status 2024-03-11T23:12:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_36 2024-03-11T23:12:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_36.status 2024-03-11T23:12:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-03-11T23:12:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:12:51.742 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:12:51.743 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:12:51.744 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:12:51.744 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:12:51.744 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:12:51.745 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:12:51.745 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 156 = '' 2024-03-11T23:12:51.745 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:12:51.745 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:12:52.206 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr:2024-03-11T23:12:52.204+0000 7fc25fa7a640 -1 reset not still connected to 0x55a8af1e81a0 2024-03-11T23:12:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_36.status 2024-03-11T23:12:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_36 2024-03-11T23:12:52.547 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_36.status 2024-03-11T23:12:52.547 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-03-11T23:12:52.547 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:12:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:12:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=156 2024-03-11T23:12:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 156 = 156 2024-03-11T23:12:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:12:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2024-03-11T23:12:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_36 2024-03-11T23:12:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:12:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:12:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:12:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:12:53.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2024-03-11T23:12:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_36': 2024-03-11T23:12:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:12:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:12:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:12:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 129710013d57 2024-03-11T23:12:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.129710013d57 2024-03-11T23:12:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:12:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:12:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:12:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:12:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:05:10 2024 2024-03-11T23:12:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:05:10 2024 2024-03-11T23:12:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:05:10 2024 2024-03-11T23:12:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:12:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:12:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:12:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 0ecf1da7-76ab-4437-ad6e-4ee82f47a299 2024-03-11T23:12:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:12:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:12:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_36 snap 2024-03-11T23:12:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_36 2024-03-11T23:12:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:12:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:12:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_36.export 2024-03-11T23:12:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_36.export 2024-03-11T23:12:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_36.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_36.export 2024-03-11T23:12:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_36.export 2024-03-11T23:12:53.826 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:12:53.832 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_36.export 2024-03-11T23:12:54.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:12:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_36.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_36.export 2024-03-11T23:12:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_36.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_36.export 2024-03-11T23:12:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:12:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:12:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_37 2024-03-11T23:12:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_37 snap 2024-03-11T23:12:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:12:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_37 2024-03-11T23:12:54.165 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:12:54.165 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:12:55.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2024-03-11T23:12:55.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:55.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:55.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_37 2024-03-11T23:12:55.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2024-03-11T23:12:55.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:55.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:55.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_37 2024-03-11T23:12:55.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:12:55.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:12:55.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:12:55.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:12:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2024-03-11T23:12:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_37 2024-03-11T23:12:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:12:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:12:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:12:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_37 2024-03-11T23:12:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:12:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:12:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:12:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:12:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:12:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:12:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:12:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:12:56.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:12:56.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:12:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:12:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2024-03-11T23:12:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:12:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:12:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:12:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:12:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:12:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:12:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:12:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:12:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:12:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:12:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:12:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:12:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:12:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:12:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:12:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:12:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:12:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:12:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:12:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:12:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:12:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:12:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:12:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:12:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:12:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:12:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:12:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:12:56.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:12:56.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:12:56.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:12:56.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2024-03-11T23:12:56.123 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-03-11T23:12:56.123 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:12:56.123 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:12:56.126 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:12:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_37 2024-03-11T23:12:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:12:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:12:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:12:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:12:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:12:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:12:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:12:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:12:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:12:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:12:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:12:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:12:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:12:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:12:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:12:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:12:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:12:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:12:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:12:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:12:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:12:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:12:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:12:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:12:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:12:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:12:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:12:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:12:56.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:12:56.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:12:56.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:12:56.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_37 2024-03-11T23:12:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-03-11T23:12:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:12:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:12:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:12:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_37 2024-03-11T23:12:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T23:12:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:12:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:12:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T23:12:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T23:12:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:12:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:12:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:12:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T23:12:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T23:12:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:12:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T23:12:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:12:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T23:12:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T23:12:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:12:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:12:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T23:12:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T23:12:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:12:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:12:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:12:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:12:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T23:12:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:12:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:12:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T23:12:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T23:12:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T23:12:56.249 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:12:56.250 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:12:56.250 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:12:56.250 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_37 2024-03-11T23:12:56.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:12:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:12:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:12:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:12:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:12:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:12:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:12:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:12:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_37 2024-03-11T23:12:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:12:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:12:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_37 2024-03-11T23:12:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:12:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:12:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_37 2024-03-11T23:12:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:12:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:12:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_37 2024-03-11T23:12:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_37.status 2024-03-11T23:12:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_37.status 2024-03-11T23:12:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:12:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_37.status 2024-03-11T23:12:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_37.status 2024-03-11T23:12:56.346 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_37.status 2024-03-11T23:12:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:12:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_37.status 2024-03-11T23:12:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_37 2024-03-11T23:12:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:12:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_37 2024-03-11T23:12:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_37 2024-03-11T23:12:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 158 2024-03-11T23:12:57.028 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_37 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_37.status 2024-03-11T23:12:57.029 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:12:57.029 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:57.029 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_37 2024-03-11T23:12:57.029 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_37.status 2024-03-11T23:12:57.029 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_37 --format xml 2024-03-11T23:12:57.029 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:12:57.074 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:12:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=158 2024-03-11T23:12:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:12:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:12:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:12:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_37.status 2024-03-11T23:12:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_37 2024-03-11T23:12:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_37.status 2024-03-11T23:12:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-03-11T23:12:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:12:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:12:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:12:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:12:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:12:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:12:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:12:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 158 = '' 2024-03-11T23:12:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:12:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:12:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_37.status 2024-03-11T23:12:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_37 2024-03-11T23:12:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_37.status 2024-03-11T23:12:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-03-11T23:12:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:12:57.778 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:12:57.778 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:12:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:12:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:12:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:12:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:12:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 158 = '' 2024-03-11T23:12:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:12:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:12:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_37.status 2024-03-11T23:12:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_37 2024-03-11T23:12:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_37.status 2024-03-11T23:12:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-03-11T23:12:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:12:58.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:12:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=158 2024-03-11T23:12:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 158 = 158 2024-03-11T23:12:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:12:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2024-03-11T23:12:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:12:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_37 2024-03-11T23:12:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:12:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:12:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:12:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:12:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2024-03-11T23:12:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_37': 2024-03-11T23:12:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:12:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:12:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:12:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 12a7cb3be7c8 2024-03-11T23:12:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.12a7cb3be7c8 2024-03-11T23:12:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:12:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:12:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:12:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:12:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:05:19 2024 2024-03-11T23:12:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:05:19 2024 2024-03-11T23:12:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:05:19 2024 2024-03-11T23:12:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:12:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:12:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:12:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 9e66f1a3-2f01-4cca-827d-d5d0c892aa22 2024-03-11T23:12:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:12:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:12:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_37 snap 2024-03-11T23:12:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:12:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_37 2024-03-11T23:12:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:12:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:12:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_37.export 2024-03-11T23:12:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_37.export 2024-03-11T23:12:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_37.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_37.export 2024-03-11T23:12:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_37.export 2024-03-11T23:12:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:12:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_37.export 2024-03-11T23:13:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:13:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_37.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_37.export 2024-03-11T23:13:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_37.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_37.export 2024-03-11T23:13:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:13:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:13:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_38 2024-03-11T23:13:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_38 snap 2024-03-11T23:13:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_38 2024-03-11T23:13:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:13:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:13:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2024-03-11T23:13:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_38 2024-03-11T23:13:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2024-03-11T23:13:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_38 2024-03-11T23:13:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:13:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:13:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:13:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:13:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2024-03-11T23:13:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_38 2024-03-11T23:13:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:13:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:13:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:13:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_38 2024-03-11T23:13:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:13:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:13:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:13:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:13:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:13:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:13:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:13:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:13:02.030 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:13:02.030 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:13:02.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:13:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2024-03-11T23:13:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:13:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:13:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:13:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:13:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:13:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:13:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:13:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:13:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:13:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:13:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:13:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:13:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:13:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:13:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:13:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:13:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:13:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:13:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:13:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:13:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:13:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:13:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:13:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:13:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:13:02.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:13:02.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:13:02.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:13:02.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:13:02.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:13:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:13:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:13:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:13:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2024-03-11T23:13:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:13:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:13:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:13:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:13:02.154 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:13:02.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:13:02.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:13:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:13:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_38 2024-03-11T23:13:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:13:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_38 2024-03-11T23:13:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:13:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:13:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_38 2024-03-11T23:13:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:13:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_38 2024-03-11T23:13:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_38.status 2024-03-11T23:13:02.157 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_38.status 2024-03-11T23:13:02.157 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:13:02.157 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_38.status 2024-03-11T23:13:02.157 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_38.status 2024-03-11T23:13:02.158 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_38.status 2024-03-11T23:13:02.158 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:13:02.158 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_38.status 2024-03-11T23:13:02.158 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_38 2024-03-11T23:13:02.159 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:02.159 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:02.159 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_38 2024-03-11T23:13:02.159 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_38 2024-03-11T23:13:03.039 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 160 2024-03-11T23:13:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_38 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_38.status 2024-03-11T23:13:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_38 2024-03-11T23:13:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_38.status 2024-03-11T23:13:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_38 --format xml 2024-03-11T23:13:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:13:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=160 2024-03-11T23:13:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:13:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:13:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:13:03.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_38.status 2024-03-11T23:13:03.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:03.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:03.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_38 2024-03-11T23:13:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_38.status 2024-03-11T23:13:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-03-11T23:13:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:03.539 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:13:03.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=72 2024-03-11T23:13:03.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 160 = 72 2024-03-11T23:13:03.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:13:03.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:13:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_38.status 2024-03-11T23:13:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_38 2024-03-11T23:13:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_38.status 2024-03-11T23:13:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-03-11T23:13:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:03.988 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:13:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:13:03.990 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:13:03.990 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:13:03.990 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:13:03.990 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:13:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 160 = '' 2024-03-11T23:13:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:13:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:13:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_38.status 2024-03-11T23:13:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_38 2024-03-11T23:13:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_38.status 2024-03-11T23:13:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-03-11T23:13:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:13:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:13:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:13:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:13:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:13:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:13:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 160 = '' 2024-03-11T23:13:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:13:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:13:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_38.status 2024-03-11T23:13:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_38 2024-03-11T23:13:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_38.status 2024-03-11T23:13:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-03-11T23:13:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:06.506 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:13:06.508 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=160 2024-03-11T23:13:06.508 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 160 = 160 2024-03-11T23:13:06.508 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:13:06.508 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2024-03-11T23:13:06.508 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_38 2024-03-11T23:13:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:13:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:13:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:13:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:13:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2024-03-11T23:13:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_38': 2024-03-11T23:13:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:13:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:13:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:13:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 12a75741f80 2024-03-11T23:13:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.12a75741f80 2024-03-11T23:13:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:13:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:13:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:13:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:13:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:05:26 2024 2024-03-11T23:13:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:05:26 2024 2024-03-11T23:13:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:05:26 2024 2024-03-11T23:13:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:13:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:13:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:13:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 0a89ddb2-e0b9-4fa8-92a7-04ff8a761c42 2024-03-11T23:13:07.555 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:13:07.557 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:13:07.557 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_38 snap 2024-03-11T23:13:07.557 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:07.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_38 2024-03-11T23:13:07.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:13:07.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:13:07.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_38.export 2024-03-11T23:13:07.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_38.export 2024-03-11T23:13:07.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_38.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_38.export 2024-03-11T23:13:07.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_38.export 2024-03-11T23:13:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:13:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_38.export 2024-03-11T23:13:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:13:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_38.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_38.export 2024-03-11T23:13:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_38.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_38.export 2024-03-11T23:13:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:13:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:13:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_39 2024-03-11T23:13:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_39 snap 2024-03-11T23:13:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_39 2024-03-11T23:13:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:13:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:13:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2024-03-11T23:13:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_39 2024-03-11T23:13:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2024-03-11T23:13:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_39 2024-03-11T23:13:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:13:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:13:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:13:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:13:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2024-03-11T23:13:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_39 2024-03-11T23:13:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:13:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:13:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:13:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_39 2024-03-11T23:13:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:13:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:13:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:13:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:13:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:13:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:13:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:13:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:13:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:13:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:13:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:13:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2024-03-11T23:13:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:13:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:13:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:13:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:13:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:13:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:13:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:13:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:13:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:13:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:13:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:13:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:13:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:13:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:13:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:13:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:13:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:13:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:13:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:13:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:13:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:13:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:13:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:13:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:13:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:13:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:13:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:13:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:13:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:13:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:13:10.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:13:10.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:13:10.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:13:10.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2024-03-11T23:13:10.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-03-11T23:13:10.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:13:10.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:13:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:13:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_39 2024-03-11T23:13:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:13:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:13:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:13:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:13:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:13:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:13:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:13:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:13:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:13:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:13:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:13:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:13:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:13:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:13:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:13:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:13:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:13:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:13:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:13:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:13:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:13:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:13:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:13:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:13:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:13:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:13:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:13:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:13:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:13:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:13:10.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:13:10.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:13:10.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:13:10.189 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_39 2024-03-11T23:13:10.281 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:13:10.281 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:13:10.281 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:13:10.282 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:13:10.282 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:13:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:13:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:13:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:13:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_39 2024-03-11T23:13:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:13:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_39 2024-03-11T23:13:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:13:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:13:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_39 2024-03-11T23:13:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:13:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_39 2024-03-11T23:13:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_39.status 2024-03-11T23:13:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_39.status 2024-03-11T23:13:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:13:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_39.status 2024-03-11T23:13:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_39.status 2024-03-11T23:13:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_39.status 2024-03-11T23:13:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:13:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_39.status 2024-03-11T23:13:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_39 2024-03-11T23:13:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_39 2024-03-11T23:13:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_39 2024-03-11T23:13:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 162 2024-03-11T23:13:11.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_39 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_39.status 2024-03-11T23:13:11.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:11.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:11.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_39 2024-03-11T23:13:11.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_39.status 2024-03-11T23:13:11.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_39 --format xml 2024-03-11T23:13:11.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:13:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=162 2024-03-11T23:13:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:13:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:13:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:13:11.591 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_39.status 2024-03-11T23:13:11.591 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:11.591 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_39 2024-03-11T23:13:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_39.status 2024-03-11T23:13:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-03-11T23:13:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:13:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=73 2024-03-11T23:13:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 162 = 73 2024-03-11T23:13:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:13:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:13:12.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_39.status 2024-03-11T23:13:12.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:12.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:12.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_39 2024-03-11T23:13:12.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_39.status 2024-03-11T23:13:12.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-03-11T23:13:12.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:12.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:13:12.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:13:12.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:13:12.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:13:12.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:13:12.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:13:12.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 162 = '' 2024-03-11T23:13:12.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:13:12.090 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:13:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_39.status 2024-03-11T23:13:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_39 2024-03-11T23:13:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_39.status 2024-03-11T23:13:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-03-11T23:13:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:12.938 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:13:12.938 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:13:12.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:13:12.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:13:12.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:13:12.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:13:12.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 162 = '' 2024-03-11T23:13:12.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:13:12.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:13:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_39.status 2024-03-11T23:13:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_39 2024-03-11T23:13:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_39.status 2024-03-11T23:13:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-03-11T23:13:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:14.593 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:13:14.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=162 2024-03-11T23:13:14.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 162 = 162 2024-03-11T23:13:14.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:13:14.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2024-03-11T23:13:14.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:14.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:14.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_39 2024-03-11T23:13:14.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:13:14.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:13:14.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:13:14.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:13:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2024-03-11T23:13:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_39': 2024-03-11T23:13:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:13:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:13:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:13:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 12a790c8580a 2024-03-11T23:13:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.12a790c8580a 2024-03-11T23:13:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:13:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:13:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:13:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:13:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:05:33 2024 2024-03-11T23:13:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:05:33 2024 2024-03-11T23:13:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:05:33 2024 2024-03-11T23:13:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:13:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:13:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:13:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: c85a3f10-e75f-4dc5-98d4-8bf7b2b4e341 2024-03-11T23:13:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:13:15.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:13:15.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_39 snap 2024-03-11T23:13:15.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:15.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_39 2024-03-11T23:13:15.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:13:15.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:13:15.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_39.export 2024-03-11T23:13:15.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_39.export 2024-03-11T23:13:15.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_39.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_39.export 2024-03-11T23:13:15.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_39.export 2024-03-11T23:13:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:13:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_39.export 2024-03-11T23:13:16.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:13:16.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_39.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_39.export 2024-03-11T23:13:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_39.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_39.export 2024-03-11T23:13:16.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:13:16.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:13:16.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_40 2024-03-11T23:13:16.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_40 snap 2024-03-11T23:13:16.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:16.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:16.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_40 2024-03-11T23:13:16.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:13:16.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:13:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2024-03-11T23:13:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_40 2024-03-11T23:13:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2024-03-11T23:13:16.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:16.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:16.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_40 2024-03-11T23:13:16.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:13:16.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:13:16.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:13:16.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:13:17.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2024-03-11T23:13:17.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:17.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:17.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_40 2024-03-11T23:13:17.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:13:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:13:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:13:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_40 2024-03-11T23:13:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:13:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:13:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:13:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:13:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:13:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:13:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:13:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:13:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:13:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:13:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:13:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2024-03-11T23:13:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:13:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:13:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:13:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:13:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:13:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:13:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:13:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:13:17.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:13:17.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:13:17.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:13:17.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:13:17.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:13:17.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:13:17.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:13:17.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:13:17.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:13:17.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:13:17.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:13:17.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:13:17.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:13:17.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:13:17.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:13:17.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:13:17.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:13:17.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:13:17.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:13:17.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:13:17.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:13:17.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:13:17.398 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:13:17.398 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:13:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:13:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2024-03-11T23:13:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:13:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:13:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:13:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:13:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:13:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:13:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:13:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:13:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_40 2024-03-11T23:13:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:13:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_40 2024-03-11T23:13:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:13:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:13:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_40 2024-03-11T23:13:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:13:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_40 2024-03-11T23:13:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_40.status 2024-03-11T23:13:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_40.status 2024-03-11T23:13:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:13:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_40.status 2024-03-11T23:13:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_40.status 2024-03-11T23:13:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_40.status 2024-03-11T23:13:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:13:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_40.status 2024-03-11T23:13:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_40 2024-03-11T23:13:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_40 2024-03-11T23:13:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_40 2024-03-11T23:13:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 164 2024-03-11T23:13:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_40 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_40.status 2024-03-11T23:13:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_40 2024-03-11T23:13:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_40.status 2024-03-11T23:13:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_40 --format xml 2024-03-11T23:13:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:13:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=164 2024-03-11T23:13:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:13:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:13:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:13:18.597 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_40.status 2024-03-11T23:13:18.597 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:18.597 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:18.597 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_40 2024-03-11T23:13:18.597 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_40.status 2024-03-11T23:13:18.597 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-03-11T23:13:18.597 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:18.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:13:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=74 2024-03-11T23:13:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 164 = 74 2024-03-11T23:13:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:13:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:13:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_40.status 2024-03-11T23:13:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_40 2024-03-11T23:13:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_40.status 2024-03-11T23:13:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-03-11T23:13:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:13:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:13:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:13:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:13:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:13:19.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:13:19.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 164 = '' 2024-03-11T23:13:19.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:13:19.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:13:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_40.status 2024-03-11T23:13:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_40 2024-03-11T23:13:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_40.status 2024-03-11T23:13:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-03-11T23:13:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:19.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:13:19.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:13:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:13:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:13:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:13:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:13:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 164 = '' 2024-03-11T23:13:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:13:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:13:21.567 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_40.status 2024-03-11T23:13:21.567 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:21.567 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:21.567 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_40 2024-03-11T23:13:21.567 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_40.status 2024-03-11T23:13:21.567 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-03-11T23:13:21.567 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:21.618 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:13:21.620 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=164 2024-03-11T23:13:21.620 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 164 = 164 2024-03-11T23:13:21.620 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:13:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2024-03-11T23:13:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_40 2024-03-11T23:13:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:13:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:13:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:13:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:13:22.621 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2024-03-11T23:13:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_40': 2024-03-11T23:13:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:13:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:13:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:13:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 12a74f20405f 2024-03-11T23:13:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.12a74f20405f 2024-03-11T23:13:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:13:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:13:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:13:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:13:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:05:47 2024 2024-03-11T23:13:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:05:47 2024 2024-03-11T23:13:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:05:47 2024 2024-03-11T23:13:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:13:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:13:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:13:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 5c2e35fd-9ddc-477b-abb2-86d6a62016b3 2024-03-11T23:13:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:13:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:13:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_40 snap 2024-03-11T23:13:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_40 2024-03-11T23:13:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:13:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:13:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_40.export 2024-03-11T23:13:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_40.export 2024-03-11T23:13:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_40.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_40.export 2024-03-11T23:13:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_40.export 2024-03-11T23:13:22.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:13:22.864 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_40.export 2024-03-11T23:13:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:13:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_40.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_40.export 2024-03-11T23:13:23.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_40.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_40.export 2024-03-11T23:13:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:13:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:13:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_41 2024-03-11T23:13:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_41 snap 2024-03-11T23:13:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_41 2024-03-11T23:13:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:13:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:13:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2024-03-11T23:13:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_41 2024-03-11T23:13:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2024-03-11T23:13:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_41 2024-03-11T23:13:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:13:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:13:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:13:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:13:24.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2024-03-11T23:13:24.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:24.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:24.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_41 2024-03-11T23:13:24.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:13:24.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:13:24.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:13:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_41 2024-03-11T23:13:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:13:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:13:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:13:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:13:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:13:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:13:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:13:24.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:13:24.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:13:24.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:13:24.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:13:24.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2024-03-11T23:13:24.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:13:24.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:13:24.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:13:24.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:13:24.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:13:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:13:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:13:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:13:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:13:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:13:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:13:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:13:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:13:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:13:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:13:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:13:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:13:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:13:24.359 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:13:24.359 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:13:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:13:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:13:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:13:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:13:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:13:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:13:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:13:24.361 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:13:24.361 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:13:24.361 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:13:24.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:13:24.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:13:24.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:13:24.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2024-03-11T23:13:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-03-11T23:13:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:13:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:13:24.463 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:13:24.463 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_41 2024-03-11T23:13:24.464 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:13:24.464 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:13:24.464 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:13:24.464 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:13:24.464 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:13:24.464 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:13:24.464 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:13:24.464 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:13:24.464 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:13:24.464 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:13:24.464 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:13:24.464 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:13:24.464 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:13:24.464 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:13:24.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:13:24.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:13:24.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:13:24.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:13:24.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:13:24.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:13:24.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:13:24.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:13:24.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:13:24.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:13:24.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:13:24.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:13:24.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:13:24.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:13:24.466 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:13:24.466 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:13:24.482 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:13:24.482 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:13:24.482 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:13:24.482 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_41 2024-03-11T23:13:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:13:24.574 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:13:24.574 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:13:24.574 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:13:24.575 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:13:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:13:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:13:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:13:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_41 2024-03-11T23:13:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:13:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_41 2024-03-11T23:13:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:13:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:13:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_41 2024-03-11T23:13:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:13:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_41 2024-03-11T23:13:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_41.status 2024-03-11T23:13:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_41.status 2024-03-11T23:13:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:13:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_41.status 2024-03-11T23:13:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_41.status 2024-03-11T23:13:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_41.status 2024-03-11T23:13:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:13:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_41.status 2024-03-11T23:13:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_41 2024-03-11T23:13:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_41 2024-03-11T23:13:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_41 2024-03-11T23:13:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 166 2024-03-11T23:13:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_41 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_41.status 2024-03-11T23:13:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_41 2024-03-11T23:13:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_41.status 2024-03-11T23:13:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_41 --format xml 2024-03-11T23:13:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:25.416 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:13:25.417 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=166 2024-03-11T23:13:25.417 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:13:25.418 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:13:25.418 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:13:25.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_41.status 2024-03-11T23:13:25.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:25.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:25.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_41 2024-03-11T23:13:25.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_41.status 2024-03-11T23:13:25.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-03-11T23:13:25.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:13:25.669 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=75 2024-03-11T23:13:25.669 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 166 = 75 2024-03-11T23:13:25.669 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:13:25.669 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:13:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_41.status 2024-03-11T23:13:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_41 2024-03-11T23:13:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_41.status 2024-03-11T23:13:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-03-11T23:13:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:26.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:13:26.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:13:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:13:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:13:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:13:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:13:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 166 = '' 2024-03-11T23:13:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:13:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:13:26.922 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_41.status 2024-03-11T23:13:26.922 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:26.922 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:26.922 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_41 2024-03-11T23:13:26.922 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_41.status 2024-03-11T23:13:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-03-11T23:13:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:13:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:13:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:13:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:13:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:13:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:13:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 166 = '' 2024-03-11T23:13:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:13:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:13:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_41.status 2024-03-11T23:13:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_41 2024-03-11T23:13:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_41.status 2024-03-11T23:13:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-03-11T23:13:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:28.634 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:13:28.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=166 2024-03-11T23:13:28.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 166 = 166 2024-03-11T23:13:28.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:13:28.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2024-03-11T23:13:28.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:28.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:28.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_41 2024-03-11T23:13:28.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:13:28.638 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:13:28.638 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:13:28.638 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:13:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2024-03-11T23:13:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_41': 2024-03-11T23:13:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:13:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:13:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:13:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 10b3f874dcb3 2024-03-11T23:13:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.10b3f874dcb3 2024-03-11T23:13:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:13:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:13:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:13:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:13:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:05:55 2024 2024-03-11T23:13:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:05:55 2024 2024-03-11T23:13:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:05:55 2024 2024-03-11T23:13:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:13:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:13:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:13:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 89993b1a-4069-4b34-96ea-7bed7be0e98e 2024-03-11T23:13:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:13:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:13:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_41 snap 2024-03-11T23:13:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_41 2024-03-11T23:13:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:13:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:13:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_41.export 2024-03-11T23:13:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_41.export 2024-03-11T23:13:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_41.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_41.export 2024-03-11T23:13:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_41.export 2024-03-11T23:13:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:13:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_41.export 2024-03-11T23:13:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:13:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_41.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_41.export 2024-03-11T23:13:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_41.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_41.export 2024-03-11T23:13:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:13:30.192 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:13:30.192 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_42 2024-03-11T23:13:30.192 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_42 snap 2024-03-11T23:13:30.192 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:30.192 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:30.192 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_42 2024-03-11T23:13:30.192 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:13:30.192 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:13:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2024-03-11T23:13:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_42 2024-03-11T23:13:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2024-03-11T23:13:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_42 2024-03-11T23:13:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:13:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:13:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:13:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:13:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2024-03-11T23:13:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_42 2024-03-11T23:13:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:13:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:13:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:13:32.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_42 2024-03-11T23:13:32.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:13:32.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:13:32.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:13:32.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:13:32.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:13:32.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:13:32.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:13:32.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:13:32.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:13:32.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:13:32.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:13:32.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2024-03-11T23:13:32.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:13:32.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:13:32.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:13:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:13:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:13:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:13:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:13:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:13:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:13:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:13:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:13:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:13:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:13:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:13:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:13:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:13:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:13:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:13:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:13:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:13:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:13:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:13:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:13:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:13:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:13:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:13:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:13:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:13:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:13:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:13:32.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:13:32.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:13:32.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:13:32.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2024-03-11T23:13:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-03-11T23:13:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:13:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:13:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:13:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_42 2024-03-11T23:13:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:13:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:13:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:13:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:13:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:13:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:13:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:13:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:13:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:13:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:13:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:13:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:13:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:13:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:13:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:13:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:13:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:13:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:13:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:13:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:13:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:13:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:13:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:13:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:13:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:13:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:13:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:13:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:13:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:13:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:13:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:13:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:13:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:13:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_42 2024-03-11T23:13:32.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:13:32.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:13:32.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:13:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:13:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:13:32.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:13:32.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:13:32.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:13:32.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_42 2024-03-11T23:13:32.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:13:32.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_42 2024-03-11T23:13:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:13:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:13:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_42 2024-03-11T23:13:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:13:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_42 2024-03-11T23:13:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_42.status 2024-03-11T23:13:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_42.status 2024-03-11T23:13:32.597 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:13:32.597 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_42.status 2024-03-11T23:13:32.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_42.status 2024-03-11T23:13:32.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_42.status 2024-03-11T23:13:32.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:13:32.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_42.status 2024-03-11T23:13:32.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_42 2024-03-11T23:13:32.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:32.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:32.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_42 2024-03-11T23:13:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_42 2024-03-11T23:13:33.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 168 2024-03-11T23:13:33.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_42 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_42.status 2024-03-11T23:13:33.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:33.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:33.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_42 2024-03-11T23:13:33.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_42.status 2024-03-11T23:13:33.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_42 --format xml 2024-03-11T23:13:33.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:13:33.423 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=168 2024-03-11T23:13:33.423 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:13:33.423 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:13:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:13:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_42.status 2024-03-11T23:13:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_42 2024-03-11T23:13:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_42.status 2024-03-11T23:13:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-03-11T23:13:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:13:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=76 2024-03-11T23:13:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 168 = 76 2024-03-11T23:13:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:13:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:13:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_42.status 2024-03-11T23:13:34.074 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:34.074 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:34.074 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_42 2024-03-11T23:13:34.074 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_42.status 2024-03-11T23:13:34.074 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-03-11T23:13:34.074 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:13:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:13:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:13:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:13:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:13:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:13:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 168 = '' 2024-03-11T23:13:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:13:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:13:34.922 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_42.status 2024-03-11T23:13:34.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:34.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:34.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_42 2024-03-11T23:13:34.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_42.status 2024-03-11T23:13:34.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-03-11T23:13:34.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:13:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:13:35.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:13:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:13:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:13:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:13:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 168 = '' 2024-03-11T23:13:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:13:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:13:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_42.status 2024-03-11T23:13:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_42 2024-03-11T23:13:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_42.status 2024-03-11T23:13:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-03-11T23:13:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:13:36.656 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=168 2024-03-11T23:13:36.656 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 168 = 168 2024-03-11T23:13:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:13:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2024-03-11T23:13:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_42 2024-03-11T23:13:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:13:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:13:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:13:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:13:37.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2024-03-11T23:13:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_42': 2024-03-11T23:13:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:13:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:13:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:13:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 11ff5ae71135 2024-03-11T23:13:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.11ff5ae71135 2024-03-11T23:13:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:13:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:13:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:13:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:13:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:06:09 2024 2024-03-11T23:13:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:06:09 2024 2024-03-11T23:13:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:06:09 2024 2024-03-11T23:13:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:13:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:13:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:13:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: f25a7039-095a-4250-884b-b71338765a8a 2024-03-11T23:13:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:13:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:13:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_42 snap 2024-03-11T23:13:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_42 2024-03-11T23:13:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:13:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:13:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_42.export 2024-03-11T23:13:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_42.export 2024-03-11T23:13:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_42.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_42.export 2024-03-11T23:13:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_42.export 2024-03-11T23:13:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:13:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_42.export 2024-03-11T23:13:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:13:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_42.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_42.export 2024-03-11T23:13:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_42.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_42.export 2024-03-11T23:13:38.220 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:13:38.221 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:13:38.221 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_43 2024-03-11T23:13:38.221 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_43 snap 2024-03-11T23:13:38.221 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:38.221 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:38.222 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_43 2024-03-11T23:13:38.222 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:13:38.222 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:13:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2024-03-11T23:13:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_43 2024-03-11T23:13:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2024-03-11T23:13:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_43 2024-03-11T23:13:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:13:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:13:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:13:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:13:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2024-03-11T23:13:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_43 2024-03-11T23:13:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:13:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:13:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:13:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_43 2024-03-11T23:13:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:13:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:13:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:13:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:13:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:13:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:13:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:13:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:13:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:13:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:13:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:13:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2024-03-11T23:13:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:13:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:13:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:13:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:13:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:13:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:13:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:13:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:13:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:13:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:13:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:13:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:13:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:13:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:13:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:13:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:13:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:13:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:13:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:13:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:13:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:13:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:13:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:13:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:13:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:13:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:13:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:13:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:13:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:13:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:13:39.407 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:13:39.408 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:13:39.408 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:13:39.408 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2024-03-11T23:13:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-03-11T23:13:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:13:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:13:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:13:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_43 2024-03-11T23:13:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:13:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:13:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:13:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:13:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:13:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:13:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:13:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:13:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:13:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:13:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:13:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:13:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:13:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:13:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:13:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:13:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:13:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:13:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:13:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:13:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:13:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:13:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:13:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:13:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:13:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:13:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:13:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:13:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:13:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:13:39.517 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:13:39.517 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:13:39.517 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:13:39.517 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_43 2024-03-11T23:13:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-03-11T23:13:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:13:39.602 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:13:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:13:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_43 2024-03-11T23:13:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T23:13:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:13:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:13:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T23:13:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T23:13:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:13:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:13:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:13:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T23:13:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T23:13:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:13:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:13:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T23:13:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:13:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T23:13:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T23:13:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:13:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:13:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T23:13:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T23:13:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:13:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:13:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:13:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:13:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T23:13:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:13:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:13:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T23:13:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T23:13:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T23:13:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:13:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:13:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:13:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_43 2024-03-11T23:13:39.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:13:39.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:13:39.713 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:13:39.713 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:13:39.713 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:13:39.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:13:39.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:13:39.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:13:39.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_43 2024-03-11T23:13:39.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:13:39.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:39.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:39.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_43 2024-03-11T23:13:39.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:13:39.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:13:39.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_43 2024-03-11T23:13:39.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:13:39.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:39.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:39.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_43 2024-03-11T23:13:39.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_43.status 2024-03-11T23:13:39.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_43.status 2024-03-11T23:13:39.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:13:39.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_43.status 2024-03-11T23:13:39.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_43.status 2024-03-11T23:13:39.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_43.status 2024-03-11T23:13:39.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:13:39.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_43.status 2024-03-11T23:13:39.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_43 2024-03-11T23:13:39.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:39.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:39.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_43 2024-03-11T23:13:39.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_43 2024-03-11T23:13:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 170 2024-03-11T23:13:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_43 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_43.status 2024-03-11T23:13:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_43 2024-03-11T23:13:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_43.status 2024-03-11T23:13:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_43 --format xml 2024-03-11T23:13:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:13:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=170 2024-03-11T23:13:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:13:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:13:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:13:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_43.status 2024-03-11T23:13:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_43 2024-03-11T23:13:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_43.status 2024-03-11T23:13:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-03-11T23:13:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:13:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=77 2024-03-11T23:13:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 170 = 77 2024-03-11T23:13:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:13:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:13:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_43.status 2024-03-11T23:13:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_43 2024-03-11T23:13:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_43.status 2024-03-11T23:13:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-03-11T23:13:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:13:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:13:41.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:13:41.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:13:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:13:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:13:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 170 = '' 2024-03-11T23:13:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:13:41.135 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:13:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_43.status 2024-03-11T23:13:41.936 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:41.936 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:41.936 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_43 2024-03-11T23:13:41.936 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_43.status 2024-03-11T23:13:41.936 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-03-11T23:13:41.936 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:13:42.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:13:42.023 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:13:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:13:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:13:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:13:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 170 = '' 2024-03-11T23:13:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:13:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:13:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_43.status 2024-03-11T23:13:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_43 2024-03-11T23:13:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_43.status 2024-03-11T23:13:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-03-11T23:13:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:13:43.682 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=170 2024-03-11T23:13:43.682 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 170 = 170 2024-03-11T23:13:43.682 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:13:43.682 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2024-03-11T23:13:43.682 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_43 2024-03-11T23:13:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:13:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:13:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:13:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:13:44.683 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2024-03-11T23:13:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_43': 2024-03-11T23:13:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:13:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:13:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:13:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 12976d2f4164 2024-03-11T23:13:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.12976d2f4164 2024-03-11T23:13:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:13:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:13:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:13:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:13:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:06:23 2024 2024-03-11T23:13:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:06:23 2024 2024-03-11T23:13:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:06:23 2024 2024-03-11T23:13:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:13:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:13:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:13:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: a46955d2-b05b-472d-bb17-68f3cabe31e2 2024-03-11T23:13:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:13:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:13:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_43 snap 2024-03-11T23:13:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_43 2024-03-11T23:13:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:13:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:13:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_43.export 2024-03-11T23:13:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_43.export 2024-03-11T23:13:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_43.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_43.export 2024-03-11T23:13:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_43.export 2024-03-11T23:13:44.909 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:13:44.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_43.export 2024-03-11T23:13:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:13:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_43.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_43.export 2024-03-11T23:13:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_43.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_43.export 2024-03-11T23:13:45.265 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:13:45.265 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:13:45.265 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_44 2024-03-11T23:13:45.265 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_44 snap 2024-03-11T23:13:45.265 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:45.265 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:45.266 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_44 2024-03-11T23:13:45.266 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:13:45.266 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:13:45.383 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2024-03-11T23:13:45.384 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:45.384 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:45.384 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_44 2024-03-11T23:13:45.384 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2024-03-11T23:13:45.384 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:45.384 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:45.384 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_44 2024-03-11T23:13:45.384 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:13:45.384 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:13:45.384 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:13:45.384 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:13:46.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2024-03-11T23:13:46.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:46.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:46.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_44 2024-03-11T23:13:46.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:13:46.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:13:46.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:13:46.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_44 2024-03-11T23:13:46.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:13:46.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:13:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:13:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:13:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:13:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:13:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:13:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:13:46.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:13:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:13:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:13:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2024-03-11T23:13:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:13:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:13:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:13:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:13:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:13:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:13:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:13:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:13:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:13:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:13:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:13:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:13:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:13:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:13:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:13:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:13:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:13:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:13:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:13:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:13:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:13:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:13:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:13:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:13:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:13:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:13:46.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:13:46.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:13:46.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:13:46.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:13:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:13:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:13:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:13:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2024-03-11T23:13:46.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-03-11T23:13:46.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:13:46.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:13:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:13:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_44 2024-03-11T23:13:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:13:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:13:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:13:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:13:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:13:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:13:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:13:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:13:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:13:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:13:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:13:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:13:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:13:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:13:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:13:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:13:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:13:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:13:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:13:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:13:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:13:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:13:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:13:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:13:46.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:13:46.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:13:46.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:13:46.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:13:46.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:13:46.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:13:46.519 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:13:46.519 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:13:46.519 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:13:46.519 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_44 2024-03-11T23:13:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-03-11T23:13:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:13:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:13:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:13:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_44 2024-03-11T23:13:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T23:13:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:13:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:13:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T23:13:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T23:13:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:13:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:13:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:13:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T23:13:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T23:13:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:13:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:13:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T23:13:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:13:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T23:13:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T23:13:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:13:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:13:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T23:13:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T23:13:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:13:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:13:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:13:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:13:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T23:13:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:13:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:13:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T23:13:46.609 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T23:13:46.609 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T23:13:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:13:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:13:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:13:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_44 2024-03-11T23:13:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:13:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:13:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:13:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:13:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:13:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:13:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:13:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:13:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_44 2024-03-11T23:13:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:13:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_44 2024-03-11T23:13:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:13:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:13:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_44 2024-03-11T23:13:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:13:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_44 2024-03-11T23:13:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_44.status 2024-03-11T23:13:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_44.status 2024-03-11T23:13:46.719 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:13:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_44.status 2024-03-11T23:13:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_44.status 2024-03-11T23:13:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_44.status 2024-03-11T23:13:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:13:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_44.status 2024-03-11T23:13:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_44 2024-03-11T23:13:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_44 2024-03-11T23:13:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_44 2024-03-11T23:13:47.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 172 2024-03-11T23:13:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_44 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_44.status 2024-03-11T23:13:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_44 2024-03-11T23:13:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_44.status 2024-03-11T23:13:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_44 --format xml 2024-03-11T23:13:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:13:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=172 2024-03-11T23:13:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:13:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:13:47.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:13:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_44.status 2024-03-11T23:13:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_44 2024-03-11T23:13:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_44.status 2024-03-11T23:13:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-03-11T23:13:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:47.707 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:13:47.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=78 2024-03-11T23:13:47.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 172 = 78 2024-03-11T23:13:47.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:13:47.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:13:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_44.status 2024-03-11T23:13:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_44 2024-03-11T23:13:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_44.status 2024-03-11T23:13:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-03-11T23:13:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:48.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:13:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:13:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:13:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:13:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:13:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:13:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 172 = '' 2024-03-11T23:13:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:13:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:13:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_44.status 2024-03-11T23:13:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_44 2024-03-11T23:13:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_44.status 2024-03-11T23:13:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-03-11T23:13:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:49.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:13:49.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:13:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:13:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:13:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:13:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:13:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 172 = '' 2024-03-11T23:13:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:13:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:13:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_44.status 2024-03-11T23:13:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_44 2024-03-11T23:13:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_44.status 2024-03-11T23:13:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-03-11T23:13:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:13:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=172 2024-03-11T23:13:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 172 = 172 2024-03-11T23:13:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:13:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2024-03-11T23:13:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_44 2024-03-11T23:13:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:13:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:13:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:13:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:13:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2024-03-11T23:13:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_44': 2024-03-11T23:13:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:13:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:13:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:13:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 12b0c66bf0c2 2024-03-11T23:13:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.12b0c66bf0c2 2024-03-11T23:13:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:13:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:13:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:13:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:13:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:06:30 2024 2024-03-11T23:13:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:06:30 2024 2024-03-11T23:13:51.741 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:06:30 2024 2024-03-11T23:13:51.741 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:13:51.741 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:13:51.741 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:13:51.741 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 5772827a-fff4-4729-bef7-03d63872ad43 2024-03-11T23:13:51.741 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:13:51.744 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:13:51.744 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_44 snap 2024-03-11T23:13:51.744 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:51.744 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_44 2024-03-11T23:13:51.744 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:13:51.744 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:13:51.744 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_44.export 2024-03-11T23:13:51.744 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_44.export 2024-03-11T23:13:51.744 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_44.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_44.export 2024-03-11T23:13:51.745 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_44.export 2024-03-11T23:13:51.922 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:13:51.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_44.export 2024-03-11T23:13:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:13:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_44.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_44.export 2024-03-11T23:13:52.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_44.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_44.export 2024-03-11T23:13:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:13:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:13:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_45 2024-03-11T23:13:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_45 snap 2024-03-11T23:13:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_45 2024-03-11T23:13:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:13:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:13:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2024-03-11T23:13:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_45 2024-03-11T23:13:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2024-03-11T23:13:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_45 2024-03-11T23:13:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:13:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:13:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:13:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:13:53.977 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2024-03-11T23:13:53.977 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:53.977 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:53.977 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_45 2024-03-11T23:13:53.977 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:13:53.977 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:13:53.977 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:13:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_45 2024-03-11T23:13:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:13:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:13:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:13:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:13:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:13:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:13:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:13:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:13:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:13:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:13:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:13:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2024-03-11T23:13:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:13:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:13:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:13:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:13:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:13:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:13:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:13:53.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:13:53.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:13:53.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:13:53.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:13:53.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:13:53.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:13:53.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:13:53.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:13:53.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:13:53.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:13:53.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:13:53.983 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:13:53.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:13:53.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:13:53.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:13:53.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:13:53.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:13:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:13:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:13:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:13:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:13:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:13:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:13:54.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:13:54.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:13:54.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:13:54.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2024-03-11T23:13:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-03-11T23:13:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:13:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:13:54.097 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:13:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_45 2024-03-11T23:13:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:13:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:13:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:13:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:13:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:13:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:13:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:13:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:13:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:13:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:13:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:13:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:13:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:13:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:13:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:13:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:13:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:13:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:13:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:13:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:13:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:13:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:13:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:13:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:13:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:13:54.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:13:54.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:13:54.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:13:54.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:13:54.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:13:54.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:13:54.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:13:54.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:13:54.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_45 2024-03-11T23:13:54.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-03-11T23:13:54.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:13:54.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:13:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:13:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_45 2024-03-11T23:13:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T23:13:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:13:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:13:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T23:13:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T23:13:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:13:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:13:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:13:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T23:13:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T23:13:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:13:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:13:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T23:13:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:13:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T23:13:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T23:13:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:13:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:13:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T23:13:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T23:13:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:13:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:13:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:13:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:13:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T23:13:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:13:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:13:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T23:13:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T23:13:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T23:13:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:13:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:13:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:13:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_45 2024-03-11T23:13:54.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:13:54.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:13:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:13:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:13:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:13:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:13:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:13:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:13:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_45 2024-03-11T23:13:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:13:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_45 2024-03-11T23:13:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:13:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:13:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_45 2024-03-11T23:13:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:13:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_45 2024-03-11T23:13:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_45.status 2024-03-11T23:13:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_45.status 2024-03-11T23:13:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:13:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_45.status 2024-03-11T23:13:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_45.status 2024-03-11T23:13:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_45.status 2024-03-11T23:13:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:13:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_45.status 2024-03-11T23:13:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_45 2024-03-11T23:13:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_45 2024-03-11T23:13:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_45 2024-03-11T23:13:54.940 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-03-11T23:13:54.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-03-11T23:13:54.941 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-03-11T23:13:54.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi132.stderr:2024-03-11T23:13:54.938+0000 7fbd2e1cb640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 62571) UID: 1000 2024-03-11T23:13:54.996 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 174 2024-03-11T23:13:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_45 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_45.status 2024-03-11T23:13:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_45 2024-03-11T23:13:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_45.status 2024-03-11T23:13:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_45 --format xml 2024-03-11T23:13:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:13:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=174 2024-03-11T23:13:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:13:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:13:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:13:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_45.status 2024-03-11T23:13:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_45 2024-03-11T23:13:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_45.status 2024-03-11T23:13:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-03-11T23:13:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:13:55.296 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=79 2024-03-11T23:13:55.296 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 174 = 79 2024-03-11T23:13:55.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:13:55.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:13:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_45.status 2024-03-11T23:13:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_45 2024-03-11T23:13:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_45.status 2024-03-11T23:13:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-03-11T23:13:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:13:55.746 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=79 2024-03-11T23:13:55.747 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 174 = 79 2024-03-11T23:13:55.747 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:13:55.747 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:13:56.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_45.status 2024-03-11T23:13:56.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:56.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:56.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_45 2024-03-11T23:13:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_45.status 2024-03-11T23:13:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-03-11T23:13:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:13:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:13:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:13:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:13:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:13:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:13:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 174 = '' 2024-03-11T23:13:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:13:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:13:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_45.status 2024-03-11T23:13:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_45 2024-03-11T23:13:58.209 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_45.status 2024-03-11T23:13:58.209 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-03-11T23:13:58.209 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:13:58.257 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:13:58.259 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=174 2024-03-11T23:13:58.259 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 174 = 174 2024-03-11T23:13:58.259 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:13:58.259 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2024-03-11T23:13:58.259 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:58.259 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:58.259 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_45 2024-03-11T23:13:58.260 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:13:58.260 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:13:58.260 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:13:58.260 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:13:58.931 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6111934913300081) > weight (0.25) 2024-03-11T23:13:58.931 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8744444942281479) > weight (0.25) 2024-03-11T23:13:58.931 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.4056787799571552) > weight (0.25) 2024-03-11T23:13:58.931 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5337538657200803) > weight (0.25) 2024-03-11T23:13:58.932 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2024-03-11T23:13:59.260 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2024-03-11T23:13:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_45': 2024-03-11T23:13:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:13:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:13:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:13:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 12b07672c923 2024-03-11T23:13:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.12b07672c923 2024-03-11T23:13:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:13:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:13:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:13:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:13:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:06:42 2024 2024-03-11T23:13:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:06:42 2024 2024-03-11T23:13:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:06:42 2024 2024-03-11T23:13:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:13:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:13:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:13:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: d3a33f3e-a3e4-4e65-b3f9-eb5141260b1a 2024-03-11T23:13:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:13:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:13:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_45 snap 2024-03-11T23:13:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_45 2024-03-11T23:13:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:13:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:13:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_45.export 2024-03-11T23:13:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_45.export 2024-03-11T23:13:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_45.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_45.export 2024-03-11T23:13:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_45.export 2024-03-11T23:13:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:13:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_45.export 2024-03-11T23:13:59.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:13:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_45.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_45.export 2024-03-11T23:13:59.817 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_45.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_45.export 2024-03-11T23:13:59.847 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:13:59.847 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:13:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_46 2024-03-11T23:13:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_46 snap 2024-03-11T23:13:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:13:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_46 2024-03-11T23:13:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:13:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:13:59.942 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-03-11T23:13:59.942 INFO:teuthology.orchestra.run:waiting for 600 2024-03-11T23:13:59.942 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-03-11T23:13:59.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-03-11T23:13:59.943 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-03-11T23:13:59.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-03-11T23:13:59.945 INFO:tasks.rbd_mirror_thrash:waiting for 77 secs before thrashing 2024-03-11T23:13:59.994 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2024-03-11T23:13:59.994 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:59.994 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_46 2024-03-11T23:13:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2024-03-11T23:13:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:13:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:13:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_46 2024-03-11T23:13:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:13:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:13:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:13:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:14:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2024-03-11T23:14:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_46 2024-03-11T23:14:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:14:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:14:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:14:00.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_46 2024-03-11T23:14:00.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:14:00.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:14:00.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:14:00.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:14:00.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:14:00.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:14:00.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:14:00.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:14:00.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:14:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:14:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:14:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2024-03-11T23:14:01.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:14:01.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:14:01.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:14:01.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:14:01.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:14:01.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:14:01.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:14:01.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:14:01.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:14:01.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:14:01.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:14:01.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:14:01.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:14:01.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:14:01.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:14:01.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:14:01.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:14:01.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:14:01.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:14:01.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:14:01.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:14:01.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:14:01.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:14:01.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:14:01.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:14:01.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:14:01.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:14:01.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:14:01.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:14:01.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:14:01.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:14:01.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:14:01.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:14:01.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2024-03-11T23:14:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-03-11T23:14:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:14:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:14:01.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:14:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_46 2024-03-11T23:14:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:14:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:14:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:14:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:14:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:14:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:14:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:14:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:14:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:14:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:14:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:14:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:14:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:14:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:14:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:14:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:14:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:14:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:14:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:14:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:14:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:14:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:14:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:14:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:14:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:14:01.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:14:01.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:14:01.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:14:01.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:14:01.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:14:01.122 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:14:01.123 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:14:01.123 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:14:01.123 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_46 2024-03-11T23:14:01.206 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-03-11T23:14:01.206 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:14:01.206 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:14:01.209 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:14:01.209 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_46 2024-03-11T23:14:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T23:14:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:14:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:14:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T23:14:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T23:14:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:14:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:14:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:14:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T23:14:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T23:14:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:14:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:14:01.211 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T23:14:01.211 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:14:01.211 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T23:14:01.211 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T23:14:01.211 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:14:01.211 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:14:01.211 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T23:14:01.211 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T23:14:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:14:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:14:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:14:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:14:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T23:14:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:14:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:14:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T23:14:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T23:14:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T23:14:01.227 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:14:01.227 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:14:01.227 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:14:01.228 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_46 2024-03-11T23:14:01.317 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:14:01.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:14:01.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:14:01.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:14:01.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:14:01.323 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:14:01.323 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:14:01.323 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:14:01.323 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_46 2024-03-11T23:14:01.323 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:14:01.323 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:01.323 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:01.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_46 2024-03-11T23:14:01.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:14:01.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:14:01.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_46 2024-03-11T23:14:01.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:14:01.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:01.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:01.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_46 2024-03-11T23:14:01.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_46.status 2024-03-11T23:14:01.324 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_46.status 2024-03-11T23:14:01.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:14:01.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_46.status 2024-03-11T23:14:01.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_46.status 2024-03-11T23:14:01.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_46.status 2024-03-11T23:14:01.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:14:01.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_46.status 2024-03-11T23:14:01.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_46 2024-03-11T23:14:01.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:01.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:01.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_46 2024-03-11T23:14:01.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_46 2024-03-11T23:14:02.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 176 2024-03-11T23:14:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_46 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_46.status 2024-03-11T23:14:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:02.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:02.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_46 2024-03-11T23:14:02.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_46.status 2024-03-11T23:14:02.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_46 --format xml 2024-03-11T23:14:02.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:14:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:14:02.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=176 2024-03-11T23:14:02.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:14:02.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:14:02.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:14:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_46.status 2024-03-11T23:14:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_46 2024-03-11T23:14:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_46.status 2024-03-11T23:14:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-03-11T23:14:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:14:02.303 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:14:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=80 2024-03-11T23:14:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 176 = 80 2024-03-11T23:14:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:14:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:14:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_46.status 2024-03-11T23:14:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_46 2024-03-11T23:14:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_46.status 2024-03-11T23:14:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-03-11T23:14:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:14:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:14:02.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=80 2024-03-11T23:14:02.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 176 = 80 2024-03-11T23:14:02.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:14:02.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:14:03.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_46.status 2024-03-11T23:14:03.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:03.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:03.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_46 2024-03-11T23:14:03.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_46.status 2024-03-11T23:14:03.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-03-11T23:14:03.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:14:03.608 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:14:03.609 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:14:03.610 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:14:03.610 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:14:03.610 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:14:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:14:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 176 = '' 2024-03-11T23:14:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:14:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:14:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_46.status 2024-03-11T23:14:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_46 2024-03-11T23:14:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_46.status 2024-03-11T23:14:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-03-11T23:14:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:14:05.263 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:14:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=176 2024-03-11T23:14:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 176 = 176 2024-03-11T23:14:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:14:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2024-03-11T23:14:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_46 2024-03-11T23:14:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:14:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:14:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:14:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:14:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2024-03-11T23:14:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_46': 2024-03-11T23:14:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:14:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:14:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:14:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 12b01e902aa4 2024-03-11T23:14:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.12b01e902aa4 2024-03-11T23:14:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:14:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:14:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:14:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:14:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:06:49 2024 2024-03-11T23:14:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:06:49 2024 2024-03-11T23:14:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:06:49 2024 2024-03-11T23:14:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:14:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:14:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:14:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: b3239d40-57ce-462a-97ff-5a0055d40fa0 2024-03-11T23:14:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:14:06.317 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:14:06.317 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_46 snap 2024-03-11T23:14:06.317 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:06.317 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_46 2024-03-11T23:14:06.317 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:14:06.317 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:14:06.317 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_46.export 2024-03-11T23:14:06.317 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_46.export 2024-03-11T23:14:06.317 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_46.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_46.export 2024-03-11T23:14:06.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_46.export 2024-03-11T23:14:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:14:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_46.export 2024-03-11T23:14:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:14:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_46.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_46.export 2024-03-11T23:14:06.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_46.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_46.export 2024-03-11T23:14:06.803 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:14:06.803 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:14:06.803 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_47 2024-03-11T23:14:06.803 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_47 snap 2024-03-11T23:14:06.803 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:06.803 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:06.803 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_47 2024-03-11T23:14:06.804 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:14:06.804 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:14:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2024-03-11T23:14:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_47 2024-03-11T23:14:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2024-03-11T23:14:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_47 2024-03-11T23:14:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:14:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:14:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:14:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:14:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2024-03-11T23:14:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_47 2024-03-11T23:14:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:14:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:14:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:14:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_47 2024-03-11T23:14:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:14:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:14:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:14:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:14:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:14:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:14:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:14:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:14:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:14:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:14:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:14:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2024-03-11T23:14:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:14:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:14:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:14:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:14:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:14:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:14:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:14:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:14:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:14:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:14:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:14:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:14:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:14:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:14:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:14:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:14:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:14:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:14:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:14:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:14:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:14:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:14:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:14:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:14:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:14:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:14:08.015 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:14:08.015 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:14:08.015 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:14:08.015 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:14:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:14:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:14:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:14:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2024-03-11T23:14:08.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-03-11T23:14:08.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:14:08.118 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:14:08.121 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:14:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_47 2024-03-11T23:14:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:14:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:14:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:14:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:14:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:14:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:14:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:14:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:14:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:14:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:14:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:14:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:14:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:14:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:14:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:14:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:14:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:14:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:14:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:14:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:14:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:14:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:14:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:14:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:14:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:14:08.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:14:08.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:14:08.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:14:08.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:14:08.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:14:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:14:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:14:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:14:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_47 2024-03-11T23:14:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:14:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:14:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:14:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:14:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:14:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:14:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:14:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:14:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_47 2024-03-11T23:14:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:14:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_47 2024-03-11T23:14:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:14:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:14:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_47 2024-03-11T23:14:08.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:14:08.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:08.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:08.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_47 2024-03-11T23:14:08.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_47.status 2024-03-11T23:14:08.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_47.status 2024-03-11T23:14:08.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:14:08.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_47.status 2024-03-11T23:14:08.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_47.status 2024-03-11T23:14:08.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_47.status 2024-03-11T23:14:08.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:14:08.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_47.status 2024-03-11T23:14:08.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_47 2024-03-11T23:14:08.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:08.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:08.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_47 2024-03-11T23:14:08.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_47 2024-03-11T23:14:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 178 2024-03-11T23:14:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_47 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_47.status 2024-03-11T23:14:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_47 2024-03-11T23:14:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_47.status 2024-03-11T23:14:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_47 --format xml 2024-03-11T23:14:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:14:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:14:09.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=178 2024-03-11T23:14:09.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:14:09.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:14:09.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:14:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_47.status 2024-03-11T23:14:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_47 2024-03-11T23:14:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_47.status 2024-03-11T23:14:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-03-11T23:14:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:14:09.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:14:09.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=81 2024-03-11T23:14:09.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 178 = 81 2024-03-11T23:14:09.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:14:09.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:14:09.721 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_47.status 2024-03-11T23:14:09.721 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:09.721 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:09.721 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_47 2024-03-11T23:14:09.721 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_47.status 2024-03-11T23:14:09.721 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-03-11T23:14:09.721 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:14:09.768 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:14:09.769 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=81 2024-03-11T23:14:09.769 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 178 = 81 2024-03-11T23:14:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:14:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:14:10.571 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_47.status 2024-03-11T23:14:10.571 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:10.571 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:10.571 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_47 2024-03-11T23:14:10.571 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_47.status 2024-03-11T23:14:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-03-11T23:14:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:14:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:14:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:14:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:14:10.626 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:14:10.626 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:14:10.626 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:14:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 178 = '' 2024-03-11T23:14:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:14:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:14:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_47.status 2024-03-11T23:14:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_47 2024-03-11T23:14:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_47.status 2024-03-11T23:14:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-03-11T23:14:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:14:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:14:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=178 2024-03-11T23:14:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 178 = 178 2024-03-11T23:14:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:14:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2024-03-11T23:14:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_47 2024-03-11T23:14:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:14:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:14:12.283 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:14:12.283 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:14:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2024-03-11T23:14:13.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_47': 2024-03-11T23:14:13.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:14:13.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:14:13.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:14:13.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 1297975de2cb 2024-03-11T23:14:13.329 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.1297975de2cb 2024-03-11T23:14:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:14:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:14:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:14:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:14:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:06:55 2024 2024-03-11T23:14:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:06:55 2024 2024-03-11T23:14:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:06:55 2024 2024-03-11T23:14:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:14:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:14:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:14:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 692e2de5-49a3-485f-a3e3-7e5e2a4ce76d 2024-03-11T23:14:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:14:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:14:13.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_47 snap 2024-03-11T23:14:13.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:13.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_47 2024-03-11T23:14:13.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:14:13.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:14:13.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_47.export 2024-03-11T23:14:13.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_47.export 2024-03-11T23:14:13.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_47.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_47.export 2024-03-11T23:14:13.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_47.export 2024-03-11T23:14:13.516 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:14:13.524 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_47.export 2024-03-11T23:14:13.770 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:14:13.777 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_47.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_47.export 2024-03-11T23:14:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_47.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_47.export 2024-03-11T23:14:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:14:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:14:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_48 2024-03-11T23:14:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_48 snap 2024-03-11T23:14:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:13.862 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:13.862 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_48 2024-03-11T23:14:13.862 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:14:13.862 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:14:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2024-03-11T23:14:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_48 2024-03-11T23:14:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2024-03-11T23:14:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_48 2024-03-11T23:14:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:14:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:14:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:14:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:14:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2024-03-11T23:14:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_48 2024-03-11T23:14:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:14:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:14:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:14:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_48 2024-03-11T23:14:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:14:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:14:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:14:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:14:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:14:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:14:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:14:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:14:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:14:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:14:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:14:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2024-03-11T23:14:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:14:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:14:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:14:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:14:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:14:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:14:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:14:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:14:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:14:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:14:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:14:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:14:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:14:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:14:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:14:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:14:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:14:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:14:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:14:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:14:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:14:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:14:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:14:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:14:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:14:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:14:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:14:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:14:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:14:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:14:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:14:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:14:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:14:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2024-03-11T23:14:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-03-11T23:14:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:14:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:14:15.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:14:15.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_48 2024-03-11T23:14:15.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:14:15.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:14:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:14:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:14:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:14:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:14:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:14:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:14:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:14:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:14:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:14:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:14:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:14:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:14:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:14:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:14:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:14:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:14:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:14:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:14:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:14:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:14:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:14:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:14:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:14:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:14:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:14:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:14:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:14:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:14:15.159 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:14:15.159 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:14:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:14:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_48 2024-03-11T23:14:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-03-11T23:14:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:14:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:14:15.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:14:15.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_48 2024-03-11T23:14:15.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T23:14:15.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:14:15.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:14:15.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T23:14:15.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T23:14:15.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:14:15.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:14:15.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:14:15.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T23:14:15.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T23:14:15.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:14:15.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:14:15.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T23:14:15.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:14:15.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T23:14:15.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T23:14:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:14:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:14:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T23:14:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T23:14:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:14:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:14:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:14:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:14:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T23:14:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:14:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:14:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T23:14:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T23:14:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T23:14:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:14:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:14:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:14:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_48 2024-03-11T23:14:15.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:14:15.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:14:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:14:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:14:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:14:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:14:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:14:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:14:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_48 2024-03-11T23:14:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:14:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_48 2024-03-11T23:14:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:14:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:14:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_48 2024-03-11T23:14:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:14:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_48 2024-03-11T23:14:15.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_48.status 2024-03-11T23:14:15.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_48.status 2024-03-11T23:14:15.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:14:15.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_48.status 2024-03-11T23:14:15.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_48.status 2024-03-11T23:14:15.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_48.status 2024-03-11T23:14:15.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:14:15.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_48.status 2024-03-11T23:14:15.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_48 2024-03-11T23:14:15.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:15.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:15.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_48 2024-03-11T23:14:15.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_48 2024-03-11T23:14:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 180 2024-03-11T23:14:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_48 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_48.status 2024-03-11T23:14:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_48 2024-03-11T23:14:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_48.status 2024-03-11T23:14:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_48 --format xml 2024-03-11T23:14:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:14:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:14:16.085 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=180 2024-03-11T23:14:16.085 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:14:16.086 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:14:16.086 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:14:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_48.status 2024-03-11T23:14:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_48 2024-03-11T23:14:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_48.status 2024-03-11T23:14:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-03-11T23:14:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:14:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:14:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=82 2024-03-11T23:14:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 180 = 82 2024-03-11T23:14:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:14:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:14:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_48.status 2024-03-11T23:14:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_48 2024-03-11T23:14:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_48.status 2024-03-11T23:14:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-03-11T23:14:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:14:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:14:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=82 2024-03-11T23:14:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 180 = 82 2024-03-11T23:14:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:14:16.787 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:14:17.588 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_48.status 2024-03-11T23:14:17.588 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:17.588 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:17.588 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_48 2024-03-11T23:14:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_48.status 2024-03-11T23:14:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-03-11T23:14:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:14:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:14:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:14:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:14:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:14:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:14:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:14:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 180 = '' 2024-03-11T23:14:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:14:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:14:19.243 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_48.status 2024-03-11T23:14:19.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:19.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:19.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_48 2024-03-11T23:14:19.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_48.status 2024-03-11T23:14:19.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-03-11T23:14:19.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:14:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:14:19.294 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=180 2024-03-11T23:14:19.294 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 180 = 180 2024-03-11T23:14:19.294 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:14:19.294 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2024-03-11T23:14:19.294 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:19.294 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:19.294 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_48 2024-03-11T23:14:19.294 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:14:19.294 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:14:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:14:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:14:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2024-03-11T23:14:20.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_48': 2024-03-11T23:14:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:14:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:14:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:14:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 12b08eca7396 2024-03-11T23:14:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.12b08eca7396 2024-03-11T23:14:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:14:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:14:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:14:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:14:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:07:07 2024 2024-03-11T23:14:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:07:07 2024 2024-03-11T23:14:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:07:07 2024 2024-03-11T23:14:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:14:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:14:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:14:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: d3f7cb01-e07f-48c5-9eba-cb65593159c2 2024-03-11T23:14:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:14:20.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:14:20.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_48 snap 2024-03-11T23:14:20.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:20.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_48 2024-03-11T23:14:20.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:14:20.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:14:20.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_48.export 2024-03-11T23:14:20.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_48.export 2024-03-11T23:14:20.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_48.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_48.export 2024-03-11T23:14:20.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_48.export 2024-03-11T23:14:20.533 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:14:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_48.export 2024-03-11T23:14:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:14:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_48.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_48.export 2024-03-11T23:14:20.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_48.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_48.export 2024-03-11T23:14:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:14:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:14:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_49 2024-03-11T23:14:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_49 snap 2024-03-11T23:14:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_49 2024-03-11T23:14:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:14:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:14:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2024-03-11T23:14:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_49 2024-03-11T23:14:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2024-03-11T23:14:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_49 2024-03-11T23:14:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:14:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:14:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:14:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:14:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2024-03-11T23:14:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_49 2024-03-11T23:14:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:14:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:14:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:14:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_49 2024-03-11T23:14:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:14:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:14:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:14:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:14:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:14:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:14:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:14:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:14:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:14:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:14:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:14:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2024-03-11T23:14:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:14:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:14:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:14:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:14:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:14:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:14:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:14:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:14:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:14:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:14:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:14:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:14:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:14:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:14:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:14:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:14:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:14:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:14:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:14:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:14:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:14:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:14:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:14:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:14:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:14:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:14:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:14:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:14:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:14:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:14:22.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:14:22.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:14:22.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:14:22.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2024-03-11T23:14:22.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-03-11T23:14:22.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:14:22.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:14:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:14:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_49 2024-03-11T23:14:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:1 2024-03-11T23:14:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:14:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:14:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-11T23:14:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:1 2024-03-11T23:14:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:14:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:14:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:14:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=1 2024-03-11T23:14:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-11T23:14:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:14:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:14:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=1 2024-03-11T23:14:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:14:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-11T23:14:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:1 2024-03-11T23:14:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:14:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:14:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-11T23:14:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:1 2024-03-11T23:14:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:14:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:14:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:14:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=1 2024-03-11T23:14:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-11T23:14:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:14:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:14:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=1 2024-03-11T23:14:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=1 2024-03-11T23:14:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-11T23:14:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:14:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:14:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-11T23:14:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_49 2024-03-11T23:14:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-03-11T23:14:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:14:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:14:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:14:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_49 2024-03-11T23:14:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:2 2024-03-11T23:14:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:14:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:14:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-03-11T23:14:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:2 2024-03-11T23:14:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:14:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:14:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:14:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=2 2024-03-11T23:14:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 2 = cluster1:2 ']' 2024-03-11T23:14:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:14:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:14:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=2 2024-03-11T23:14:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:14:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-03-11T23:14:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:2 2024-03-11T23:14:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:14:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:14:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-03-11T23:14:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:2 2024-03-11T23:14:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:14:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:14:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:14:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=2 2024-03-11T23:14:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 2 = cluster1:2 ']' 2024-03-11T23:14:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:14:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:14:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=2 2024-03-11T23:14:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=2 2024-03-11T23:14:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-03-11T23:14:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:14:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:14:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-03-11T23:14:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_49 2024-03-11T23:14:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-03-11T23:14:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd mirror status : get status for rbd mirror 2024-03-11T23:14:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:admin_socket: invalid command 2024-03-11T23:14:22.346 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:14:22.346 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_49 2024-03-11T23:14:22.346 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:3 2024-03-11T23:14:22.346 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:14:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:14:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-03-11T23:14:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:3 2024-03-11T23:14:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:14:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:14:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:14:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=3 2024-03-11T23:14:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 3 = cluster1:3 ']' 2024-03-11T23:14:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:14:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:14:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=3 2024-03-11T23:14:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:14:22.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-03-11T23:14:22.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:3 2024-03-11T23:14:22.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:14:22.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:14:22.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-03-11T23:14:22.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:3 2024-03-11T23:14:22.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:14:22.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:14:22.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:14:22.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=3 2024-03-11T23:14:22.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 3 = cluster1:3 ']' 2024-03-11T23:14:22.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:14:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:14:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=3 2024-03-11T23:14:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=3 2024-03-11T23:14:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-03-11T23:14:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:14:22.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:14:22.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-03-11T23:14:22.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_49 2024-03-11T23:14:22.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:14:22.455 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:14:22.455 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:14:22.455 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:14:22.455 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:14:22.457 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:14:22.457 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:14:22.457 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:14:22.457 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_49 2024-03-11T23:14:22.457 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:14:22.457 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:22.457 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:22.457 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_49 2024-03-11T23:14:22.457 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:14:22.457 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:14:22.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_49 2024-03-11T23:14:22.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:14:22.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:22.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:22.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_49 2024-03-11T23:14:22.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_49.status 2024-03-11T23:14:22.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_49.status 2024-03-11T23:14:22.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:14:22.459 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_49.status 2024-03-11T23:14:22.459 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_49.status 2024-03-11T23:14:22.459 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_49.status 2024-03-11T23:14:22.459 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:14:22.460 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_49.status 2024-03-11T23:14:22.460 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_49 2024-03-11T23:14:22.460 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:22.460 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:22.460 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_49 2024-03-11T23:14:22.460 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_49 2024-03-11T23:14:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 182 2024-03-11T23:14:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_49 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_49.status 2024-03-11T23:14:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_49 2024-03-11T23:14:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_49.status 2024-03-11T23:14:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_49 --format xml 2024-03-11T23:14:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:14:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:14:23.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=182 2024-03-11T23:14:23.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:14:23.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:14:23.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:14:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_49.status 2024-03-11T23:14:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_49 2024-03-11T23:14:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_49.status 2024-03-11T23:14:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-03-11T23:14:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:14:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:14:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=83 2024-03-11T23:14:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 182 = 83 2024-03-11T23:14:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:14:23.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:14:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_49.status 2024-03-11T23:14:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_49 2024-03-11T23:14:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_49.status 2024-03-11T23:14:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-03-11T23:14:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:14:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:14:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=83 2024-03-11T23:14:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 182 = 83 2024-03-11T23:14:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:14:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:14:24.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_49.status 2024-03-11T23:14:24.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:24.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:24.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_49 2024-03-11T23:14:24.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_49.status 2024-03-11T23:14:24.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-03-11T23:14:24.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:14:24.652 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:14:24.652 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:14:24.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:14:24.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:14:24.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:14:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:14:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 182 = '' 2024-03-11T23:14:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:14:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:14:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_49.status 2024-03-11T23:14:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_49 2024-03-11T23:14:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_49.status 2024-03-11T23:14:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-03-11T23:14:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:14:26.306 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:14:26.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=182 2024-03-11T23:14:26.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 182 = 182 2024-03-11T23:14:26.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:14:26.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2024-03-11T23:14:26.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:26.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:26.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_49 2024-03-11T23:14:26.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:14:26.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:14:26.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:14:26.308 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:14:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2024-03-11T23:14:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_49': 2024-03-11T23:14:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:14:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:14:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:14:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 10b3118e0451 2024-03-11T23:14:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.10b3118e0451 2024-03-11T23:14:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:14:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:14:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:14:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:14:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:07:21 2024 2024-03-11T23:14:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:07:21 2024 2024-03-11T23:14:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:07:21 2024 2024-03-11T23:14:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:14:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:14:27.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:14:27.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 71ab5b7e-440a-48ab-8f45-bb970a72ca7e 2024-03-11T23:14:27.355 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:14:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:14:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_49 snap 2024-03-11T23:14:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_49 2024-03-11T23:14:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:14:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:14:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_49.export 2024-03-11T23:14:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_49.export 2024-03-11T23:14:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_49.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_49.export 2024-03-11T23:14:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_49.export 2024-03-11T23:14:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:14:27.545 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_49.export 2024-03-11T23:14:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:14:27.752 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_49.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_49.export 2024-03-11T23:14:27.806 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_49.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_49.export 2024-03-11T23:14:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:14:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:14:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_50 2024-03-11T23:14:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_snap cluster2 mirror image_50 snap 2024-03-11T23:14:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_50 2024-03-11T23:14:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:14:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2024-03-11T23:14:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2024-03-11T23:14:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_50 2024-03-11T23:14:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2024-03-11T23:14:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_50 2024-03-11T23:14:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=started 2024-03-11T23:14:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:14:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:14:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:14:29.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2024-03-11T23:14:29.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:29.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:29.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_50 2024-03-11T23:14:29.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=started 2024-03-11T23:14:29.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_result 2024-03-11T23:14:29.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=stopped 2024-03-11T23:14:29.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_50 2024-03-11T23:14:29.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_instance=cluster1 2024-03-11T23:14:29.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep -i state 2024-03-11T23:14:29.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:14:29.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:14:29.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance=cluster1 2024-03-11T23:14:29.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local loop_instance 2024-03-11T23:14:29.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-11T23:14:29.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sleep 0 2024-03-11T23:14:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-11T23:14:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ seq 0 3 2024-03-11T23:14:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-11T23:14:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2024-03-11T23:14:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1:0 2024-03-11T23:14:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ shift 2024-03-11T23:14:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance 2024-03-11T23:14:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-11T23:14:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local val=cluster1:0 2024-03-11T23:14:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster_var_name=cluster 2024-03-11T23:14:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local instance_var_name=instance 2024-03-11T23:14:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:14:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:14:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-11T23:14:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval cluster=cluster1 2024-03-11T23:14:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:14:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ eval instance=0 2024-03-11T23:14:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:14:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-11T23:14:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local local_cluster=cluster1:0 2024-03-11T23:14:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster=cluster1 2024-03-11T23:14:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance 2024-03-11T23:14:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-11T23:14:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local val=cluster1:0 2024-03-11T23:14:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local cluster_var_name=local_cluster 2024-03-11T23:14:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ local instance_var_name=instance 2024-03-11T23:14:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ cluster=cluster1 2024-03-11T23:14:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ instance=0 2024-03-11T23:14:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-11T23:14:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval local_cluster=cluster1 2024-03-11T23:14:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ local_cluster=cluster1 2024-03-11T23:14:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ eval instance=0 2024-03-11T23:14:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ instance=0 2024-03-11T23:14:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-11T23:14:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:14:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:14:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-11T23:14:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2024-03-11T23:14:29.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ return 0 2024-03-11T23:14:29.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ status_result=' "state": "Replaying"' 2024-03-11T23:14:29.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo ' "state": "Replaying"' 2024-03-11T23:14:29.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ grep -i Replaying 2024-03-11T23:14:29.172 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: "state": "Replaying" 2024-03-11T23:14:29.173 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=started 2024-03-11T23:14:29.173 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test started = started 2024-03-11T23:14:29.174 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:14:29.174 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_50 2024-03-11T23:14:29.174 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:14:29.174 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:29.174 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:29.174 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_50 2024-03-11T23:14:29.174 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = journal ']' 2024-03-11T23:14:29.174 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' snapshot = snapshot ']' 2024-03-11T23:14:29.174 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_50 2024-03-11T23:14:29.174 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_cluster=cluster1 2024-03-11T23:14:29.174 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_50 2024-03-11T23:14:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster2-mirror-image_50.status 2024-03-11T23:14:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster2-mirror-image_50.status 2024-03-11T23:14:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:14:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local status_log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_50.status 2024-03-11T23:14:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ mkfname cluster1-mirror-image_50.status 2024-03-11T23:14:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo cluster1-mirror-image_50.status 2024-03-11T23:14:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-11T23:14:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local local_status_log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_50.status 2024-03-11T23:14:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ mirror_image_snapshot cluster2 mirror image_50 2024-03-11T23:14:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_50 2024-03-11T23:14:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_50 2024-03-11T23:14:30.059 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Snapshot ID: 184 2024-03-11T23:14:30.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_50 /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_50.status 2024-03-11T23:14:30.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:30.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:30.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_50 2024-03-11T23:14:30.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_50.status 2024-03-11T23:14:30.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_50 --format xml 2024-03-11T23:14:30.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:14:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-03-11T23:14:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snapshot_id=184 2024-03-11T23:14:30.111 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:14:30.111 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:14:30.111 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.2 2024-03-11T23:14:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_50.status 2024-03-11T23:14:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_50 2024-03-11T23:14:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_50.status 2024-03-11T23:14:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-03-11T23:14:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:14:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:14:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=84 2024-03-11T23:14:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 184 = 84 2024-03-11T23:14:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:14:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.4 2024-03-11T23:14:30.761 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_50.status 2024-03-11T23:14:30.761 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_50 2024-03-11T23:14:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_50.status 2024-03-11T23:14:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-03-11T23:14:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:14:30.810 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:14:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=84 2024-03-11T23:14:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 184 = 84 2024-03-11T23:14:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:14:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.8 2024-03-11T23:14:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_50.status 2024-03-11T23:14:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_50 2024-03-11T23:14:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_50.status 2024-03-11T23:14:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-03-11T23:14:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:14:31.663 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:14:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:14:31.665 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:-:1.1: Document is empty 2024-03-11T23:14:31.665 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: 2024-03-11T23:14:31.665 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:^ 2024-03-11T23:14:31.665 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id= 2024-03-11T23:14:31.666 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 184 = '' 2024-03-11T23:14:31.666 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-11T23:14:31.666 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1.6 2024-03-11T23:14:33.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_50.status 2024-03-11T23:14:33.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:33.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:33.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_50 2024-03-11T23:14:33.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local log=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_50.status 2024-03-11T23:14:33.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-03-11T23:14:33.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-03-11T23:14:33.316 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-03-11T23:14:33.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local primary_snapshot_id=184 2024-03-11T23:14:33.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 184 = 184 2024-03-11T23:14:33.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:14:33.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2024-03-11T23:14:33.318 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:14:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_50 2024-03-11T23:14:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:14:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:14:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-11T23:14:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:14:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2024-03-11T23:14:34.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_50': 2024-03-11T23:14:34.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 128 MiB in 32 objects 2024-03-11T23:14:34.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:14:34.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 2 2024-03-11T23:14:34.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 11ff159d33a5 2024-03-11T23:14:34.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.11ff159d33a5 2024-03-11T23:14:34.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:14:34.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering, non-primary 2024-03-11T23:14:34.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:14:34.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:14:34.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:07:28 2024 2024-03-11T23:14:34.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:07:28 2024 2024-03-11T23:14:34.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:07:28 2024 2024-03-11T23:14:34.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: protected: False 2024-03-11T23:14:34.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:14:34.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:14:34.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 3fb7dde7-de15-40cd-9bc8-e3db123cf169 2024-03-11T23:14:34.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: false 2024-03-11T23:14:34.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:14:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror image_50 snap 2024-03-11T23:14:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_50 2024-03-11T23:14:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap_name=snap 2024-03-11T23:14:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local ret=0 2024-03-11T23:14:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local rmt_export=/tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_50.export 2024-03-11T23:14:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local loc_export=/tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_50.export 2024-03-11T23:14:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_50.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_50.export 2024-03-11T23:14:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_50.export 2024-03-11T23:14:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:14:34.556 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_50.export 2024-03-11T23:14:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-11T23:14:34.767 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cmp /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_50.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_50.export 2024-03-11T23:14:34.821 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rm -f /tmp/tmp.7mIVT0l9FS/cluster2-mirror-image_50.export /tmp/tmp.7mIVT0l9FS/cluster1-mirror-image_50.export 2024-03-11T23:14:34.852 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:14:34.852 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ testlog 'TEST: delete many images' 2024-03-11T23:14:34.852 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ tee -a /tmp/tmp.7mIVT0l9FS/rbd-mirror.test.log 2024-03-11T23:14:34.852 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ date '+%F %T' 2024-03-11T23:14:34.852 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo 2024-03-11 23:14:34 TEST: delete many images 2024-03-11T23:14:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:2024-03-11 23:14:34 TEST: delete many images 2024-03-11T23:14:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ seq 1 50 2024-03-11T23:14:34.854 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:14:34.854 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_1 2024-03-11T23:14:34.854 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2024-03-11T23:14:34.854 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T23:14:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:14:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2024-03-11T23:14:35.075 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:14:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_1 2024-03-11T23:14:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T23:14:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:14:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:14:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_1 2024-03-11T23:14:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T23:14:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-03-11T23:14:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-03-11T23:14:35.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi132.stderr:2024-03-11T23:14:35.221+0000 7fac0f35f640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5629d3789ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:14:35.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi132.stderr:2024-03-11T23:14:35.221+0000 7fac15b6c640 -1 rbd::mirror::ImageReplayer: 0x5629d3628c80 [3/7380f9c0-c386-40d3-8842-2968a362a7ba] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:14:36.163 INFO:tasks.ceph.cluster2.osd.2.smithi132.stderr:2024-03-11T23:14:36.160+0000 7f6376022640 -1 reset not still connected to 0x565098912a90 2024-03-11T23:14:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:14:36.197 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:14:36.197 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:14:36.197 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_2 2024-03-11T23:14:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2024-03-11T23:14:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T23:14:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:14:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2024-03-11T23:14:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:14:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_2 2024-03-11T23:14:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T23:14:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:14:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:14:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_2 2024-03-11T23:14:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T23:14:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-03-11T23:14:37.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-03-11T23:14:37.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr:2024-03-11T23:14:37.234+0000 7f6de8915640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555be6096b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:14:37.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr:2024-03-11T23:14:37.234+0000 7f6dee921640 -1 rbd::mirror::ImageReplayer: 0x555be6818c80 [3/0db5ab54-08c1-4a9b-b8f1-d2ac90c55bee] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:14:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:14:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:14:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:14:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_3 2024-03-11T23:14:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2024-03-11T23:14:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T23:14:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:14:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2024-03-11T23:14:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:14:39.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_3 2024-03-11T23:14:39.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:39.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:39.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T23:14:39.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:14:39.101 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:14:39.101 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_3 2024-03-11T23:14:39.101 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:39.101 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T23:14:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-03-11T23:14:39.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-03-11T23:14:39.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi132.stderr:2024-03-11T23:14:39.247+0000 7fac0f35f640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5629d3789860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:14:39.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi132.stderr:2024-03-11T23:14:39.247+0000 7fac15b6c640 -1 rbd::mirror::ImageReplayer: 0x5629d3628f00 [3/89903746-8b7b-4dea-b492-96c6f9aebf71] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:14:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:14:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:14:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:14:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_4 2024-03-11T23:14:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2024-03-11T23:14:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T23:14:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:14:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2024-03-11T23:14:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:14:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_4 2024-03-11T23:14:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T23:14:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:14:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:14:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_4 2024-03-11T23:14:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T23:14:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-03-11T23:14:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-03-11T23:14:41.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi132.stderr:2024-03-11T23:14:41.250+0000 7f4ea480a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55666237b520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:14:41.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi132.stderr:2024-03-11T23:14:41.250+0000 7f4eaa816640 -1 rbd::mirror::ImageReplayer: 0x556662312f00 [3/fc8576a3-dbba-4c16-95dd-f75d6193b45b] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:14:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:14:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:14:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:14:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_5 2024-03-11T23:14:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2024-03-11T23:14:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T23:14:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:14:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2024-03-11T23:14:43.114 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:14:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_5 2024-03-11T23:14:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T23:14:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:14:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:14:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_5 2024-03-11T23:14:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T23:14:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-03-11T23:14:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-03-11T23:14:43.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi132.stderr:2024-03-11T23:14:43.264+0000 7f1e2d08c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557c7c39f380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:14:43.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi132.stderr:2024-03-11T23:14:43.264+0000 7f1e3409a640 -1 rbd::mirror::ImageReplayer: 0x557c7cc31400 [3/6e7c5b9b-f490-4f68-8ae3-de136df2e981] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:14:44.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:14:44.248 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:14:44.248 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:14:44.248 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_6 2024-03-11T23:14:44.248 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2024-03-11T23:14:44.248 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:44.248 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:44.249 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:14:44.249 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:14:44.249 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2024-03-11T23:14:44.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:14:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_6 2024-03-11T23:14:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:14:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:14:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:14:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_6 2024-03-11T23:14:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:14:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-03-11T23:14:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-03-11T23:14:44.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi132.stderr:2024-03-11T23:14:44.488+0000 7f4ea480a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55666237b860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:14:44.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi132.stderr:2024-03-11T23:14:44.488+0000 7f4eab017640 -1 rbd::mirror::ImageReplayer: 0x556662282280 [3/fdce66ca-291f-45dc-b094-a088660b842f] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:14:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:14:45.466 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:14:45.466 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:14:45.466 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_7 2024-03-11T23:14:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2024-03-11T23:14:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:14:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:14:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2024-03-11T23:14:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:14:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_7 2024-03-11T23:14:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:14:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:14:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:14:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_7 2024-03-11T23:14:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:14:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-03-11T23:14:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-03-11T23:14:46.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi132.stderr:2024-03-11T23:14:46.511+0000 7fac0f35f640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5629d2760820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:14:46.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi132.stderr:2024-03-11T23:14:46.511+0000 7fac1636d640 -1 rbd::mirror::ImageReplayer: 0x5629d3629400 [3/e7d22bec-3c5d-4227-9c8e-5fb052d5be91] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:14:47.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:14:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:14:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:14:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_8 2024-03-11T23:14:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2024-03-11T23:14:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_8 2024-03-11T23:14:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:14:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2024-03-11T23:14:48.353 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:14:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_8 2024-03-11T23:14:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_8 2024-03-11T23:14:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:14:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:14:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_8 2024-03-11T23:14:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_8 2024-03-11T23:14:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-03-11T23:14:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-03-11T23:14:48.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi132.stderr:2024-03-11T23:14:48.525+0000 7fac0fb60640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5629d2f02ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:14:48.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi132.stderr:2024-03-11T23:14:48.525+0000 7fac1636d640 -1 rbd::mirror::ImageReplayer: 0x5629d25da000 [3/ac93902e-f852-49af-ac1f-b13ea2b5269b] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:14:49.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:14:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:14:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:14:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_9 2024-03-11T23:14:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2024-03-11T23:14:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_9 2024-03-11T23:14:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:14:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2024-03-11T23:14:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:14:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_9 2024-03-11T23:14:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_9 2024-03-11T23:14:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:14:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:14:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_9 2024-03-11T23:14:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_9 2024-03-11T23:14:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-03-11T23:14:50.424 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-03-11T23:14:50.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr:2024-03-11T23:14:50.522+0000 7f6de8915640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555be5fa0ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:14:50.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr:2024-03-11T23:14:50.522+0000 7f6dee921640 -1 rbd::mirror::ImageReplayer: 0x555be68ca000 [3/b02ac2fb-bf75-42d9-bf3f-a8404d4fe797] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:14:51.480 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:14:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:14:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:14:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_10 2024-03-11T23:14:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2024-03-11T23:14:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_10 2024-03-11T23:14:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:14:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2024-03-11T23:14:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:14:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_10 2024-03-11T23:14:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_10 2024-03-11T23:14:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:14:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:14:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_10 2024-03-11T23:14:52.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:52.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:52.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_10 2024-03-11T23:14:52.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-03-11T23:14:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-03-11T23:14:52.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr:2024-03-11T23:14:52.536+0000 7f6de8114640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555be67e0d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:14:52.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr:2024-03-11T23:14:52.536+0000 7f6def122640 -1 rbd::mirror::ImageReplayer: 0x555be6818780 [3/6e7e7d8e-c0d0-4968-90fa-7731ffa367f1] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:14:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:14:53.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:14:53.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:14:53.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_11 2024-03-11T23:14:53.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2024-03-11T23:14:53.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:53.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:53.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_11 2024-03-11T23:14:53.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:14:53.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2024-03-11T23:14:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:14:54.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_11 2024-03-11T23:14:54.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:54.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:54.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_11 2024-03-11T23:14:54.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:14:54.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:14:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_11 2024-03-11T23:14:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_11 2024-03-11T23:14:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2024-03-11T23:14:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2024-03-11T23:14:54.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi132.stderr:2024-03-11T23:14:54.511+0000 7f4ea480a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55666237a680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:14:54.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi132.stderr:2024-03-11T23:14:54.511+0000 7f4eaa816640 -1 rbd::mirror::ImageReplayer: 0x556662282a00 [3/952b2674-c455-44d6-b523-5d182beac4cc] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:14:55.404 INFO:tasks.ceph.cluster2.osd.2.smithi132.stderr:2024-03-11T23:14:55.399+0000 7f6376022640 -1 reset not still connected to 0x565089dc61a0 2024-03-11T23:14:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:14:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:14:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:14:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_12 2024-03-11T23:14:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2024-03-11T23:14:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_12 2024-03-11T23:14:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:14:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2024-03-11T23:14:56.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:14:56.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_12 2024-03-11T23:14:56.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:56.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:56.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_12 2024-03-11T23:14:56.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:14:56.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:14:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_12 2024-03-11T23:14:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_12 2024-03-11T23:14:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2024-03-11T23:14:56.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2024-03-11T23:14:56.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi132.stderr:2024-03-11T23:14:56.552+0000 7fac0f35f640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5629d2760ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:14:56.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi132.stderr:2024-03-11T23:14:56.552+0000 7fac1636d640 -1 rbd::mirror::ImageReplayer: 0x5629d3376c80 [3/f99801f5-5b32-4c9d-b6d4-607f161fc716] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:14:57.511 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:14:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:14:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:14:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_13 2024-03-11T23:14:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2024-03-11T23:14:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_13 2024-03-11T23:14:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:14:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2024-03-11T23:14:58.406 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:14:58.418 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_13 2024-03-11T23:14:58.418 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:58.418 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:58.418 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_13 2024-03-11T23:14:58.418 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:14:58.418 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:14:58.419 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_13 2024-03-11T23:14:58.419 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:58.419 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:58.419 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_13 2024-03-11T23:14:58.420 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2024-03-11T23:14:58.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2024-03-11T23:14:58.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi132.stderr:2024-03-11T23:14:58.563+0000 7f1e2d88d640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557c7c39f1e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:14:58.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi132.stderr:2024-03-11T23:14:58.563+0000 7f1e3409a640 -1 rbd::mirror::ImageReplayer: 0x557c7cc30c80 [3/4e5cb2c2-5e2b-44b7-95cc-f0928b52a2e0] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:14:59.466 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:14:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:14:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:14:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_14 2024-03-11T23:14:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2024-03-11T23:14:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:14:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:14:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_14 2024-03-11T23:14:59.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:14:59.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2024-03-11T23:15:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:15:00.421 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_14 2024-03-11T23:15:00.421 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_14 2024-03-11T23:15:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:15:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:15:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_14 2024-03-11T23:15:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_14 2024-03-11T23:15:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2024-03-11T23:15:00.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2024-03-11T23:15:00.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi132.stderr:2024-03-11T23:15:00.560+0000 7f4ea4009640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55666237ba00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:15:00.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi132.stderr:2024-03-11T23:15:00.560+0000 7f4eab017640 -1 rbd::mirror::ImageReplayer: 0x556662282780 [3/e8312f3c-c1b7-4832-92dc-03abff4ac406] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:15:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:15:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:15:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:15:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_15 2024-03-11T23:15:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2024-03-11T23:15:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_15 2024-03-11T23:15:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:15:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2024-03-11T23:15:02.409 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:15:02.420 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_15 2024-03-11T23:15:02.420 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:02.420 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:02.420 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_15 2024-03-11T23:15:02.420 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:15:02.420 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:15:02.422 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_15 2024-03-11T23:15:02.422 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:02.422 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:02.422 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_15 2024-03-11T23:15:02.422 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2024-03-11T23:15:02.468 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2024-03-11T23:15:02.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi132.stderr:2024-03-11T23:15:02.571+0000 7fac0fb60640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5629d2f02680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:15:02.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi132.stderr:2024-03-11T23:15:02.571+0000 7fac15b6c640 -1 rbd::mirror::ImageReplayer: 0x5629d3629180 [3/bfe4c714-e6df-48ee-b9ba-69d0b7a5f35b] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:15:03.526 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:15:03.529 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:15:03.529 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:15:03.529 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_16 2024-03-11T23:15:03.529 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2024-03-11T23:15:03.529 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:03.529 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:03.529 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_16 2024-03-11T23:15:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:15:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2024-03-11T23:15:04.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:15:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_16 2024-03-11T23:15:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_16 2024-03-11T23:15:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:15:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:15:04.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_16 2024-03-11T23:15:04.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:04.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:04.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_16 2024-03-11T23:15:04.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2024-03-11T23:15:04.409 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2024-03-11T23:15:04.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr:2024-03-11T23:15:04.500+0000 7f6de8114640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555be67e11e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:15:04.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr:2024-03-11T23:15:04.500+0000 7f6def122640 -1 rbd::mirror::ImageReplayer: 0x555be6819b80 [3/8afe84ee-3bfc-43a7-b04d-7925c8799653] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:15:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:15:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:15:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:15:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_17 2024-03-11T23:15:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2024-03-11T23:15:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:05.556 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_17 2024-03-11T23:15:05.556 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:15:05.556 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2024-03-11T23:15:06.444 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:15:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_17 2024-03-11T23:15:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_17 2024-03-11T23:15:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:15:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:15:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_17 2024-03-11T23:15:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:06.455 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:06.455 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_17 2024-03-11T23:15:06.455 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2024-03-11T23:15:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2024-03-11T23:15:06.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi132.stderr:2024-03-11T23:15:06.591+0000 7f1e2d88d640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557c7c3a0340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:15:06.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi132.stderr:2024-03-11T23:15:06.591+0000 7f1e33899640 -1 rbd::mirror::ImageReplayer: 0x557c7cc30500 [3/2a3f0f67-66fa-49bb-9d97-83ed1ad32bc2] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:15:07.570 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:15:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:15:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:15:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_18 2024-03-11T23:15:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2024-03-11T23:15:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_18 2024-03-11T23:15:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:15:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2024-03-11T23:15:08.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:15:08.464 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_18 2024-03-11T23:15:08.464 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:08.464 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:08.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_18 2024-03-11T23:15:08.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:15:08.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:15:08.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_18 2024-03-11T23:15:08.466 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:08.466 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:08.466 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_18 2024-03-11T23:15:08.466 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2024-03-11T23:15:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2024-03-11T23:15:08.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr:2024-03-11T23:15:08.608+0000 7f6de8915640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555be5fa0000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:15:08.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr:2024-03-11T23:15:08.608+0000 7f6dee921640 -1 rbd::mirror::ImageReplayer: 0x555be6818000 [3/387adbd8-be3f-401f-ba8e-7062c5a54dfd] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:15:09.459 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:15:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:15:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:15:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_19 2024-03-11T23:15:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2024-03-11T23:15:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_19 2024-03-11T23:15:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:15:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2024-03-11T23:15:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:15:10.470 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_19 2024-03-11T23:15:10.470 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:10.470 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:10.470 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_19 2024-03-11T23:15:10.470 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:15:10.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:15:10.472 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_19 2024-03-11T23:15:10.472 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:10.472 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:10.472 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_19 2024-03-11T23:15:10.472 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2024-03-11T23:15:10.526 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2024-03-11T23:15:10.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi132.stderr:2024-03-11T23:15:10.628+0000 7f4ea4009640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55666237bba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:15:10.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi132.stderr:2024-03-11T23:15:10.628+0000 7f4eab017640 -1 rbd::mirror::ImageReplayer: 0x556662282f00 [3/06685589-ab88-4e65-be67-9272ee49920c] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:15:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:15:11.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:15:11.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:15:11.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_20 2024-03-11T23:15:11.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2024-03-11T23:15:11.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_20 2024-03-11T23:15:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:15:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2024-03-11T23:15:12.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:15:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_20 2024-03-11T23:15:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_20 2024-03-11T23:15:12.490 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:15:12.490 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:15:12.490 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_20 2024-03-11T23:15:12.490 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:12.490 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:12.490 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_20 2024-03-11T23:15:12.490 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2024-03-11T23:15:12.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2024-03-11T23:15:12.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi132.stderr:2024-03-11T23:15:12.636+0000 7f4ea480a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556661ac7380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:15:12.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi132.stderr:2024-03-11T23:15:12.636+0000 7f4eaa816640 -1 rbd::mirror::ImageReplayer: 0x556662282000 [3/7b95035b-6c99-40a9-947e-f623904f126b] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:15:13.591 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:15:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:15:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:15:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_21 2024-03-11T23:15:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2024-03-11T23:15:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_21 2024-03-11T23:15:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:15:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2024-03-11T23:15:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:15:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_21 2024-03-11T23:15:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_21 2024-03-11T23:15:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:15:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:15:14.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_21 2024-03-11T23:15:14.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:14.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:14.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_21 2024-03-11T23:15:14.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2024-03-11T23:15:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2024-03-11T23:15:14.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi132.stderr:2024-03-11T23:15:14.729+0000 7f1e2d88d640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557c7c3a01a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:15:14.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi132.stderr:2024-03-11T23:15:14.729+0000 7f1e33899640 -1 rbd::mirror::ImageReplayer: 0x557c7cc30a00 [3/32652090-5ba2-4262-a3f3-521e042b1019] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:15:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:15:15.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:15:15.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:15:15.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_22 2024-03-11T23:15:15.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2024-03-11T23:15:15.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:15.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_22 2024-03-11T23:15:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:15:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2024-03-11T23:15:16.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:15:16.509 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_22 2024-03-11T23:15:16.509 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:16.509 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:16.510 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_22 2024-03-11T23:15:16.510 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:15:16.510 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:15:16.510 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_22 2024-03-11T23:15:16.510 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:16.510 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:16.511 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_22 2024-03-11T23:15:16.511 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2024-03-11T23:15:16.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2024-03-11T23:15:16.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi132.stderr:2024-03-11T23:15:16.648+0000 7f1e2d08c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557c7c3a1380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:15:16.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi132.stderr:2024-03-11T23:15:16.648+0000 7f1e33899640 -1 rbd::mirror::ImageReplayer: 0x557c7cc31180 [3/6b2da8c4-e5c3-4ec9-acc6-083f710996ff] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:15:16.946 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.754819983098302) > weight (0.25) 2024-03-11T23:15:16.946 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-03-11T23:15:16.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-03-11T23:15:16.947 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-03-11T23:15:16.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr:2024-03-11T23:15:16.944+0000 7fa08ecb7640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 46030) UID: 1000 2024-03-11T23:15:17.608 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:15:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:15:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:15:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_23 2024-03-11T23:15:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2024-03-11T23:15:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_23 2024-03-11T23:15:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:15:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2024-03-11T23:15:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:15:18.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_23 2024-03-11T23:15:18.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:18.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:18.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_23 2024-03-11T23:15:18.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:15:18.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:15:18.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_23 2024-03-11T23:15:18.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:18.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:18.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_23 2024-03-11T23:15:18.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2024-03-11T23:15:18.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2024-03-11T23:15:18.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi132.stderr:2024-03-11T23:15:18.642+0000 7f1e2d08c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557c7c3a0ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:15:18.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi132.stderr:2024-03-11T23:15:18.642+0000 7f1e33899640 -1 rbd::mirror::ImageReplayer: 0x557c7cc31680 [3/6f278f19-20af-4e77-81d1-1dfe17bc389b] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:15:18.948 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-03-11T23:15:18.948 INFO:teuthology.orchestra.run:waiting for 600 2024-03-11T23:15:18.948 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-03-11T23:15:18.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-03-11T23:15:18.949 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-03-11T23:15:18.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-03-11T23:15:18.951 INFO:tasks.rbd_mirror_thrash:waiting for 86 secs before thrashing 2024-03-11T23:15:19.464 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:15:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:15:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:15:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_24 2024-03-11T23:15:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2024-03-11T23:15:19.468 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:19.468 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:19.468 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_24 2024-03-11T23:15:19.468 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:15:19.468 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2024-03-11T23:15:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:15:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_24 2024-03-11T23:15:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_24 2024-03-11T23:15:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:15:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:15:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_24 2024-03-11T23:15:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_24 2024-03-11T23:15:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2024-03-11T23:15:20.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2024-03-11T23:15:20.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi132.stderr:2024-03-11T23:15:20.617+0000 7f84c2717640 -1 rbd::mirror::ImageMap: 0x55ba0ebfaea0 handle_peer_ack_remove: failed to remove global_image_id=edbba68c-29ef-4285-9e1b-327ac0002f9c 2024-03-11T23:15:21.620 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:15:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:15:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:15:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_25 2024-03-11T23:15:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2024-03-11T23:15:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_25 2024-03-11T23:15:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:15:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2024-03-11T23:15:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:15:22.517 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_25 2024-03-11T23:15:22.518 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:22.518 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:22.518 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_25 2024-03-11T23:15:22.518 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:15:22.518 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:15:22.518 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_25 2024-03-11T23:15:22.518 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:22.518 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:22.518 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_25 2024-03-11T23:15:22.518 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2024-03-11T23:15:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2024-03-11T23:15:22.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi132.stderr:2024-03-11T23:15:22.670+0000 7f1e2d88d640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557c7c39f040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:15:22.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi132.stderr:2024-03-11T23:15:22.670+0000 7f1e3409a640 -1 rbd::mirror::ImageReplayer: 0x557c7ccd2000 [3/d58404ce-fb8c-4239-ba38-049eab7f80aa] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:15:23.626 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:15:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:15:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:15:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_26 2024-03-11T23:15:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2024-03-11T23:15:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_26 2024-03-11T23:15:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:15:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2024-03-11T23:15:24.345 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:15:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_26 2024-03-11T23:15:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_26 2024-03-11T23:15:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:15:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:15:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_26 2024-03-11T23:15:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_26 2024-03-11T23:15:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2024-03-11T23:15:24.416 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2024-03-11T23:15:24.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi132.stderr:2024-03-11T23:15:24.467+0000 7f84c2717640 -1 rbd::mirror::ImageMap: 0x55ba0ebfaea0 handle_peer_ack_remove: failed to remove global_image_id=d759a618-182e-4efe-82fa-def08c221d57 2024-03-11T23:15:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:15:25.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:15:25.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:15:25.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_27 2024-03-11T23:15:25.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2024-03-11T23:15:25.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:25.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:25.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_27 2024-03-11T23:15:25.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:15:25.477 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2024-03-11T23:15:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:15:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_27 2024-03-11T23:15:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_27 2024-03-11T23:15:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:15:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:15:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_27 2024-03-11T23:15:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_27 2024-03-11T23:15:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2024-03-11T23:15:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2024-03-11T23:15:26.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi132.stderr:2024-03-11T23:15:26.511+0000 7fac0f35f640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5629d2f031e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:15:26.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi132.stderr:2024-03-11T23:15:26.511+0000 7fac15b6c640 -1 rbd::mirror::ImageReplayer: 0x5629d3377180 [3/f55425aa-fe4e-4d48-89cf-caafa49c5423] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:15:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:15:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:15:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:15:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_28 2024-03-11T23:15:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2024-03-11T23:15:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_28 2024-03-11T23:15:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:15:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2024-03-11T23:15:28.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:15:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_28 2024-03-11T23:15:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_28 2024-03-11T23:15:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:15:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:15:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_28 2024-03-11T23:15:28.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:28.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:28.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_28 2024-03-11T23:15:28.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2024-03-11T23:15:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2024-03-11T23:15:28.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi132.stderr:2024-03-11T23:15:28.478+0000 7f84c2717640 -1 rbd::mirror::ImageMap: 0x55ba0ebfaea0 handle_peer_ack_remove: failed to remove global_image_id=93b4a8e6-8dee-44ce-b32e-2d1497b47e9c 2024-03-11T23:15:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:15:29.480 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:15:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:15:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_29 2024-03-11T23:15:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2024-03-11T23:15:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_29 2024-03-11T23:15:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:15:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2024-03-11T23:15:30.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:15:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_29 2024-03-11T23:15:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_29 2024-03-11T23:15:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:15:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:15:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_29 2024-03-11T23:15:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_29 2024-03-11T23:15:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2024-03-11T23:15:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2024-03-11T23:15:30.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr:2024-03-11T23:15:30.545+0000 7f6de8915640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555be5fa0b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:15:30.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr:2024-03-11T23:15:30.545+0000 7f6dee921640 -1 rbd::mirror::ImageReplayer: 0x555be6819400 [3/56839585-8992-435c-80a3-a4d4b2cb652c] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:15:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:15:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:15:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:15:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_30 2024-03-11T23:15:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2024-03-11T23:15:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_30 2024-03-11T23:15:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:15:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2024-03-11T23:15:32.377 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:15:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_30 2024-03-11T23:15:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_30 2024-03-11T23:15:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:15:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:15:32.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_30 2024-03-11T23:15:32.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:32.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:32.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_30 2024-03-11T23:15:32.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2024-03-11T23:15:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2024-03-11T23:15:32.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi132.stderr:2024-03-11T23:15:32.536+0000 7f4ea4009640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556661ac2d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:15:32.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi132.stderr:2024-03-11T23:15:32.536+0000 7f4eaa816640 -1 rbd::mirror::ImageReplayer: 0x556662283680 [3/5b5325b7-4350-47ba-9791-39702d0ec94a] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:15:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:15:33.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:15:33.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:15:33.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_31 2024-03-11T23:15:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2024-03-11T23:15:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_31 2024-03-11T23:15:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:15:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2024-03-11T23:15:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:15:34.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_31 2024-03-11T23:15:34.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:34.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:34.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_31 2024-03-11T23:15:34.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:15:34.365 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:15:34.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_31 2024-03-11T23:15:34.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:34.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:34.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_31 2024-03-11T23:15:34.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2024-03-11T23:15:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2024-03-11T23:15:34.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi132.stderr:2024-03-11T23:15:34.519+0000 7f4ea4009640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5566619f5380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:15:34.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi132.stderr:2024-03-11T23:15:34.519+0000 7f4eab017640 -1 rbd::mirror::ImageReplayer: 0x556662283400 [3/0fbab2fa-8751-419f-a846-c51d8786c5c5] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:15:34.932 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.2745312310739234) > weight (0.25) 2024-03-11T23:15:34.932 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4427974021400457) > weight (0.25) 2024-03-11T23:15:34.932 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7846460574218761) > weight (0.25) 2024-03-11T23:15:34.932 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.7517056407994207) > weight (0.25) 2024-03-11T23:15:34.933 INFO:tasks.rbd_mirror_thrash:waiting for 98 secs before thrashing 2024-03-11T23:15:35.522 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:15:35.526 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:15:35.526 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:15:35.526 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_32 2024-03-11T23:15:35.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2024-03-11T23:15:35.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:35.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:35.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_32 2024-03-11T23:15:35.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:15:35.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2024-03-11T23:15:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:15:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_32 2024-03-11T23:15:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_32 2024-03-11T23:15:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:15:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:15:36.407 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_32 2024-03-11T23:15:36.407 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:36.407 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:36.407 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_32 2024-03-11T23:15:36.407 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2024-03-11T23:15:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2024-03-11T23:15:36.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi132.stderr:2024-03-11T23:15:36.522+0000 7f84c1f16640 -1 rbd::mirror::ImageMap: 0x55ba0ebfaea0 handle_peer_ack_remove: failed to remove global_image_id=adeee68d-39bd-4c47-9b90-d83d5ac98497 2024-03-11T23:15:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:15:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:15:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:15:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_33 2024-03-11T23:15:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2024-03-11T23:15:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_33 2024-03-11T23:15:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:15:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2024-03-11T23:15:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:15:38.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_33 2024-03-11T23:15:38.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:38.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:38.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_33 2024-03-11T23:15:38.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:15:38.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:15:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_33 2024-03-11T23:15:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_33 2024-03-11T23:15:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2024-03-11T23:15:38.481 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2024-03-11T23:15:38.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi132.stderr:2024-03-11T23:15:38.577+0000 7fac0f35f640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5629d2f03380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:15:38.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi132.stderr:2024-03-11T23:15:38.577+0000 7fac1636d640 -1 rbd::mirror::ImageReplayer: 0x5629d3628000 [3/94a5319a-2e74-4f65-a321-3f962cd4ebe5] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:15:38.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi132.stderr:2024-03-11T23:15:38.788+0000 7fac0fb60640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5629d2f03ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:15:38.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi132.stderr:2024-03-11T23:15:38.788+0000 7fac15b6c640 -1 rbd::mirror::ImageReplayer: 0x5629d3628000 [3/94a5319a-2e74-4f65-a321-3f962cd4ebe5] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:15:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:15:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:15:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:15:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_34 2024-03-11T23:15:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2024-03-11T23:15:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_34 2024-03-11T23:15:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:15:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2024-03-11T23:15:40.415 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:15:40.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_34 2024-03-11T23:15:40.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:40.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:40.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_34 2024-03-11T23:15:40.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:15:40.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:15:40.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_34 2024-03-11T23:15:40.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:40.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_34 2024-03-11T23:15:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2024-03-11T23:15:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2024-03-11T23:15:40.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi132.stderr:2024-03-11T23:15:40.536+0000 7f84c2717640 -1 rbd::mirror::ImageMap: 0x55ba0ebfaea0 handle_peer_ack_remove: failed to remove global_image_id=fa15ec82-36d2-421d-b90c-12034ab39d68 2024-03-11T23:15:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:15:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:15:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:15:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_35 2024-03-11T23:15:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2024-03-11T23:15:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_35 2024-03-11T23:15:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:15:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2024-03-11T23:15:42.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:15:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_35 2024-03-11T23:15:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_35 2024-03-11T23:15:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:15:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:15:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_35 2024-03-11T23:15:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_35 2024-03-11T23:15:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2024-03-11T23:15:42.490 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2024-03-11T23:15:42.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi132.stderr:2024-03-11T23:15:42.588+0000 7fac0f35f640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5629d2e22820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:15:42.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi132.stderr:2024-03-11T23:15:42.588+0000 7fac1636d640 -1 rbd::mirror::ImageReplayer: 0x5629d3376780 [3/ddb7a161-de34-428e-ae0c-47939be794d1] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:15:43.545 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:15:43.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:15:43.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:15:43.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_36 2024-03-11T23:15:43.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2024-03-11T23:15:43.550 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:43.550 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:43.550 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_36 2024-03-11T23:15:43.550 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:15:43.550 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2024-03-11T23:15:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:15:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_36 2024-03-11T23:15:44.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:44.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:44.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_36 2024-03-11T23:15:44.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:15:44.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:15:44.359 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_36 2024-03-11T23:15:44.359 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:44.359 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:44.359 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_36 2024-03-11T23:15:44.359 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2024-03-11T23:15:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2024-03-11T23:15:44.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr:2024-03-11T23:15:44.504+0000 7f6de8114640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555be5fa2ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:15:44.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr:2024-03-11T23:15:44.504+0000 7f6def122640 -1 rbd::mirror::ImageReplayer: 0x555be6818f00 [3/0ecf1da7-76ab-4437-ad6e-4ee82f47a299] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:15:45.560 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:15:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:15:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:15:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_37 2024-03-11T23:15:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2024-03-11T23:15:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_37 2024-03-11T23:15:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:15:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2024-03-11T23:15:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:15:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_37 2024-03-11T23:15:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_37 2024-03-11T23:15:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:15:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:15:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_37 2024-03-11T23:15:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_37 2024-03-11T23:15:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2024-03-11T23:15:46.480 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2024-03-11T23:15:46.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi132.stderr:2024-03-11T23:15:46.589+0000 7f1e2d88d640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557c7c3a0000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:15:46.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi132.stderr:2024-03-11T23:15:46.589+0000 7f1e33899640 -1 rbd::mirror::ImageReplayer: 0x557c7cc30280 [3/9e66f1a3-2f01-4cca-827d-d5d0c892aa22] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:15:47.559 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:15:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:15:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:15:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_38 2024-03-11T23:15:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2024-03-11T23:15:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_38 2024-03-11T23:15:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:15:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2024-03-11T23:15:48.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:15:48.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_38 2024-03-11T23:15:48.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:48.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:48.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_38 2024-03-11T23:15:48.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:15:48.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:15:48.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_38 2024-03-11T23:15:48.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:48.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:48.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_38 2024-03-11T23:15:48.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2024-03-11T23:15:48.492 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2024-03-11T23:15:48.580 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2024-03-11T23:15:48.576+0000 7ff0c0b41640 -1 reset not still connected to 0x558839fb15f0 2024-03-11T23:15:48.580 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2024-03-11T23:15:48.576+0000 7ff0c0b41640 -1 reset not still connected to 0x55883a7fedd0 2024-03-11T23:15:48.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi132.stderr:2024-03-11T23:15:48.597+0000 7f4ea480a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5566619f36c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:15:48.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi132.stderr:2024-03-11T23:15:48.597+0000 7f4eab017640 -1 rbd::mirror::ImageReplayer: 0x556662283180 [3/0a89ddb2-e0b9-4fa8-92a7-04ff8a761c42] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:15:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:15:49.481 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:15:49.481 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:15:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_39 2024-03-11T23:15:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2024-03-11T23:15:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_39 2024-03-11T23:15:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:15:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2024-03-11T23:15:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:15:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_39 2024-03-11T23:15:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_39 2024-03-11T23:15:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:15:50.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:15:50.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_39 2024-03-11T23:15:50.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:50.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:50.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_39 2024-03-11T23:15:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2024-03-11T23:15:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2024-03-11T23:15:50.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi132.stderr:2024-03-11T23:15:50.599+0000 7fac0fb60640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5629d2e23d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:15:50.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi132.stderr:2024-03-11T23:15:50.599+0000 7fac15b6c640 -1 rbd::mirror::ImageReplayer: 0x5629d3376000 [3/c85a3f10-e75f-4dc5-98d4-8bf7b2b4e341] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:15:51.573 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:15:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:15:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:15:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_40 2024-03-11T23:15:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2024-03-11T23:15:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_40 2024-03-11T23:15:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:15:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2024-03-11T23:15:52.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:15:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_40 2024-03-11T23:15:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_40 2024-03-11T23:15:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:15:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:15:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_40 2024-03-11T23:15:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_40 2024-03-11T23:15:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2024-03-11T23:15:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2024-03-11T23:15:52.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr:2024-03-11T23:15:52.605+0000 7f6de8114640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555be5fa3380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:15:52.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr:2024-03-11T23:15:52.605+0000 7f6def122640 -1 rbd::mirror::ImageReplayer: 0x555be6819900 [3/5c2e35fd-9ddc-477b-abb2-86d6a62016b3] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:15:53.578 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:15:53.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:15:53.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:15:53.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_41 2024-03-11T23:15:53.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2024-03-11T23:15:53.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:53.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:53.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_41 2024-03-11T23:15:53.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:15:53.580 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2024-03-11T23:15:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:15:54.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_41 2024-03-11T23:15:54.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:54.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:54.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_41 2024-03-11T23:15:54.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:15:54.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:15:54.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_41 2024-03-11T23:15:54.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:54.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:54.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_41 2024-03-11T23:15:54.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2024-03-11T23:15:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2024-03-11T23:15:54.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi132.stderr:2024-03-11T23:15:54.521+0000 7f1e2d08c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557c7c3a11e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:15:54.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi132.stderr:2024-03-11T23:15:54.521+0000 7f1e33899640 -1 rbd::mirror::ImageReplayer: 0x557c7cc30000 [3/89993b1a-4069-4b34-96ea-7bed7be0e98e] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:15:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:15:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:15:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:15:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_42 2024-03-11T23:15:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2024-03-11T23:15:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_42 2024-03-11T23:15:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:15:55.593 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2024-03-11T23:15:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:15:56.464 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr:2024-03-11T23:15:56.461+0000 7f4dc2287640 -1 reset not still connected to 0x55f1a6a8a340 2024-03-11T23:15:56.473 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_42 2024-03-11T23:15:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_42 2024-03-11T23:15:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:15:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:15:56.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_42 2024-03-11T23:15:56.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_42 2024-03-11T23:15:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2024-03-11T23:15:56.522 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2024-03-11T23:15:56.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi132.stderr:2024-03-11T23:15:56.625+0000 7fac0fb60640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5629d2e23860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:15:56.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi132.stderr:2024-03-11T23:15:56.625+0000 7fac1636d640 -1 rbd::mirror::ImageReplayer: 0x5629d3376500 [3/f25a7039-095a-4250-884b-b71338765a8a] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:15:57.589 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:15:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:15:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:15:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_43 2024-03-11T23:15:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2024-03-11T23:15:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_43 2024-03-11T23:15:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:15:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2024-03-11T23:15:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:15:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_43 2024-03-11T23:15:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_43 2024-03-11T23:15:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:15:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:15:58.498 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_43 2024-03-11T23:15:58.498 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:58.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:58.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_43 2024-03-11T23:15:58.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2024-03-11T23:15:58.545 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2024-03-11T23:15:58.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi132.stderr:2024-03-11T23:15:58.646+0000 7f1e2d88d640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557c7c3a1380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:15:58.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi132.stderr:2024-03-11T23:15:58.646+0000 7f1e3409a640 -1 rbd::mirror::ImageReplayer: 0x557c7cc31b80 [3/a46955d2-b05b-472d-bb17-68f3cabe31e2] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:15:59.484 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:15:59.487 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:15:59.487 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:15:59.487 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_44 2024-03-11T23:15:59.487 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2024-03-11T23:15:59.488 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:15:59.488 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:15:59.488 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_44 2024-03-11T23:15:59.488 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:15:59.488 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2024-03-11T23:16:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:16:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_44 2024-03-11T23:16:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:16:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_44 2024-03-11T23:16:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:16:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:16:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_44 2024-03-11T23:16:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:16:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_44 2024-03-11T23:16:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2024-03-11T23:16:00.550 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2024-03-11T23:16:00.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr:2024-03-11T23:16:00.657+0000 7f6de8915640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555be5fa31e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:16:00.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr:2024-03-11T23:16:00.657+0000 7f6def122640 -1 rbd::mirror::ImageReplayer: 0x555be6819680 [3/5772827a-fff4-4729-bef7-03d63872ad43] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:16:01.610 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:16:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:01.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_45 2024-03-11T23:16:01.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2024-03-11T23:16:01.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:16:01.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:01.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_45 2024-03-11T23:16:01.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:16:01.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2024-03-11T23:16:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:16:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_45 2024-03-11T23:16:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:16:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_45 2024-03-11T23:16:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:16:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:16:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_45 2024-03-11T23:16:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:16:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_45 2024-03-11T23:16:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2024-03-11T23:16:02.563 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2024-03-11T23:16:02.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr:2024-03-11T23:16:02.668+0000 7f6de8114640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555be6097040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:16:02.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr:2024-03-11T23:16:02.668+0000 7f6def122640 -1 rbd::mirror::ImageReplayer: 0x555be6818a00 [3/d3a33f3e-a3e4-4e65-b3f9-eb5141260b1a] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:16:03.623 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:16:03.626 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:03.626 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:03.626 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_46 2024-03-11T23:16:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2024-03-11T23:16:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:16:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_46 2024-03-11T23:16:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:16:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2024-03-11T23:16:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:16:04.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_46 2024-03-11T23:16:04.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:16:04.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:04.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_46 2024-03-11T23:16:04.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:16:04.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:16:04.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_46 2024-03-11T23:16:04.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:16:04.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:04.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_46 2024-03-11T23:16:04.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2024-03-11T23:16:04.419 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2024-03-11T23:16:04.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi132.stderr:2024-03-11T23:16:04.517+0000 7f4ea4009640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556661ac2ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:16:04.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi132.stderr:2024-03-11T23:16:04.517+0000 7f4eaa816640 -1 rbd::mirror::ImageReplayer: 0x556662283b80 [3/b3239d40-57ce-462a-97ff-5a0055d40fa0] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:16:05.832 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:16:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_47 2024-03-11T23:16:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2024-03-11T23:16:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:16:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_47 2024-03-11T23:16:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:16:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2024-03-11T23:16:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:16:06.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_47 2024-03-11T23:16:06.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:16:06.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:06.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_47 2024-03-11T23:16:06.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:16:06.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:16:06.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_47 2024-03-11T23:16:06.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:16:06.528 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:06.528 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_47 2024-03-11T23:16:06.528 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2024-03-11T23:16:06.576 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2024-03-11T23:16:06.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi132.stderr:2024-03-11T23:16:06.679+0000 7f1e2d88d640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557c7c3a0340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:16:06.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi132.stderr:2024-03-11T23:16:06.679+0000 7f1e3409a640 -1 rbd::mirror::ImageReplayer: 0x557c7cc30f00 [3/692e2de5-49a3-485f-a3e3-7e5e2a4ce76d] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:16:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:16:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_48 2024-03-11T23:16:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2024-03-11T23:16:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:16:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_48 2024-03-11T23:16:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:16:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2024-03-11T23:16:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:16:08.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_48 2024-03-11T23:16:08.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:16:08.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:08.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_48 2024-03-11T23:16:08.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:16:08.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:16:08.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_48 2024-03-11T23:16:08.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:16:08.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:08.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_48 2024-03-11T23:16:08.541 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2024-03-11T23:16:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2024-03-11T23:16:08.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi132.stderr:2024-03-11T23:16:08.686+0000 7fac0fb60640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5629d2e236c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:16:08.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi132.stderr:2024-03-11T23:16:08.686+0000 7fac1636d640 -1 rbd::mirror::ImageReplayer: 0x5629d3376f00 [3/d3f7cb01-e07f-48c5-9eba-cb65593159c2] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:16:08.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi132.stderr:2024-03-11T23:16:08.787+0000 7fac0fb60640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5629d2e22680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:16:08.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi132.stderr:2024-03-11T23:16:08.787+0000 7fac15b6c640 -1 rbd::mirror::ImageReplayer: 0x5629d3376f00 [3/d3f7cb01-e07f-48c5-9eba-cb65593159c2] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:16:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:16:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_49 2024-03-11T23:16:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2024-03-11T23:16:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:16:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_49 2024-03-11T23:16:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:16:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2024-03-11T23:16:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:16:10.548 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_49 2024-03-11T23:16:10.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:16:10.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:10.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_49 2024-03-11T23:16:10.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:16:10.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:16:10.550 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_49 2024-03-11T23:16:10.550 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:16:10.550 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:10.550 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_49 2024-03-11T23:16:10.550 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2024-03-11T23:16:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2024-03-11T23:16:10.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr:2024-03-11T23:16:10.706+0000 7f6de8915640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555be6099a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:16:10.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr:2024-03-11T23:16:10.706+0000 7f6dee921640 -1 rbd::mirror::ImageReplayer: 0x555be6818500 [3/71ab5b7e-440a-48ab-8f45-bb970a72ca7e] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:16:11.672 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:16:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_50 2024-03-11T23:16:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2024-03-11T23:16:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:16:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_50 2024-03-11T23:16:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local snap=snap 2024-03-11T23:16:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2024-03-11T23:16:12.542 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing snap: 100% complete...done. 2024-03-11T23:16:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_50 2024-03-11T23:16:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:16:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_50 2024-03-11T23:16:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:16:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:16:12.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_50 2024-03-11T23:16:12.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:16:12.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:12.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_50 2024-03-11T23:16:12.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2024-03-11T23:16:12.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2024-03-11T23:16:12.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr:2024-03-11T23:16:12.708+0000 7f6de8114640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555be6099ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:16:12.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr:2024-03-11T23:16:12.708+0000 7f6dee921640 -1 rbd::mirror::ImageReplayer: 0x555be6818280 [3/3fb7dde7-de15-40cd-9bc8-e3db123cf169] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:16:13.666 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-11T23:16:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ testlog 'TEST: image deletions should propagate' 2024-03-11T23:16:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ tee -a /tmp/tmp.7mIVT0l9FS/rbd-mirror.test.log 2024-03-11T23:16:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ date '+%F %T' 2024-03-11T23:16:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo 2024-03-11 23:16:13 TEST: image deletions should propagate 2024-03-11T23:16:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:2024-03-11 23:16:13 TEST: image deletions should propagate 2024-03-11T23:16:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_pool_images cluster1 mirror 0 2024-03-11T23:16:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_count=0 2024-03-11T23:16:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local count 2024-03-11T23:16:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local last_count=0 2024-03-11T23:16:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ true 2024-03-11T23:16:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ seq 1 40 2024-03-11T23:16:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in `seq 1 40` 2024-03-11T23:16:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 1 -ne 1 2024-03-11T23:16:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-03-11T23:16:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep 'images: ' 2024-03-11T23:16:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cut '-d ' -f 2 2024-03-11T23:16:13.707 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: failed to query services: (13) Permission denied 2024-03-11T23:16:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ count=0 2024-03-11T23:16:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 0 = 0 2024-03-11T23:16:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2024-03-11T23:16:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state 2024-03-11T23:16:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ seq 1 40 2024-03-11T23:16:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in `seq 1 40` 2024-03-11T23:16:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test 1 -ne 1 2024-03-11T23:16:13.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-03-11T23:16:13.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ grep 'image health:' 2024-03-11T23:16:13.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cut '-d ' -f 3 2024-03-11T23:16:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: failed to query services: (13) Permission denied 2024-03-11T23:16:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ state=OK 2024-03-11T23:16:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test OK = ERROR 2024-03-11T23:16:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test OK = OK 2024-03-11T23:16:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ seq 1 50 2024-03-11T23:16:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_1 2024-03-11T23:16:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-03-11T23:16:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T23:16:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_1 2024-03-11T23:16:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_1 2024-03-11T23:16:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-03-11T23:16:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-03-11T23:16:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-03-11T23:16:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T23:16:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_1 2024-03-11T23:16:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_1 2024-03-11T23:16:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-03-11T23:16:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:13.919 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-03-11T23:16:13.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:13.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:13.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:13.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:13.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:13.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_2 2024-03-11T23:16:13.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-03-11T23:16:13.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T23:16:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_2 2024-03-11T23:16:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_2 2024-03-11T23:16:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-03-11T23:16:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-03-11T23:16:13.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:13.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:13.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-03-11T23:16:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T23:16:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_2 2024-03-11T23:16:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_2 2024-03-11T23:16:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-03-11T23:16:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:14.092 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-03-11T23:16:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_3 2024-03-11T23:16:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-03-11T23:16:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T23:16:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_3 2024-03-11T23:16:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_3 2024-03-11T23:16:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-03-11T23:16:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-03-11T23:16:14.129 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:14.130 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:14.130 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:14.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-03-11T23:16:14.231 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T23:16:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_3 2024-03-11T23:16:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_3 2024-03-11T23:16:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-03-11T23:16:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-03-11T23:16:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_4 2024-03-11T23:16:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-03-11T23:16:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T23:16:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_4 2024-03-11T23:16:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_4 2024-03-11T23:16:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-03-11T23:16:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:14.298 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-03-11T23:16:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-03-11T23:16:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T23:16:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_4 2024-03-11T23:16:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_4 2024-03-11T23:16:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-03-11T23:16:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-03-11T23:16:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_5 2024-03-11T23:16:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-03-11T23:16:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T23:16:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_5 2024-03-11T23:16:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_5 2024-03-11T23:16:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-03-11T23:16:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:14.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-03-11T23:16:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:14.574 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-03-11T23:16:14.574 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:14.574 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:14.574 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T23:16:14.574 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:14.574 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_5 2024-03-11T23:16:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_5 2024-03-11T23:16:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-03-11T23:16:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-03-11T23:16:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_6 2024-03-11T23:16:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-03-11T23:16:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:16:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_6 2024-03-11T23:16:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_6 2024-03-11T23:16:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-03-11T23:16:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-03-11T23:16:14.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-03-11T23:16:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:16:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_6 2024-03-11T23:16:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_6 2024-03-11T23:16:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-03-11T23:16:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-03-11T23:16:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_7 2024-03-11T23:16:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-03-11T23:16:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:16:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_7 2024-03-11T23:16:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_7 2024-03-11T23:16:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-03-11T23:16:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:14.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-03-11T23:16:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:14.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-03-11T23:16:14.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:14.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:14.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:16:14.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:14.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:14.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:14.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:14.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_7 2024-03-11T23:16:14.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:14.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_7 2024-03-11T23:16:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-03-11T23:16:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:14.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-03-11T23:16:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:14.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_8 2024-03-11T23:16:14.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-03-11T23:16:14.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:14.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:14.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_8 2024-03-11T23:16:14.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:14.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:14.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:14.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:14.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_8 2024-03-11T23:16:14.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:14.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:14.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_8 2024-03-11T23:16:14.957 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-03-11T23:16:14.957 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-03-11T23:16:14.988 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-03-11T23:16:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_8 2024-03-11T23:16:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_8 2024-03-11T23:16:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_8 2024-03-11T23:16:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-03-11T23:16:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:15.123 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-03-11T23:16:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_9 2024-03-11T23:16:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-03-11T23:16:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_9 2024-03-11T23:16:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_9 2024-03-11T23:16:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_9 2024-03-11T23:16:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-03-11T23:16:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:15.159 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-03-11T23:16:15.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:15.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:15.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:15.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-03-11T23:16:15.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:15.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:15.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_9 2024-03-11T23:16:15.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:15.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:15.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:15.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:15.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_9 2024-03-11T23:16:15.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:15.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:15.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_9 2024-03-11T23:16:15.265 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-03-11T23:16:15.265 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-03-11T23:16:15.299 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_10 2024-03-11T23:16:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-03-11T23:16:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_10 2024-03-11T23:16:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_10 2024-03-11T23:16:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_10 2024-03-11T23:16:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-03-11T23:16:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:15.331 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-03-11T23:16:15.335 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:15.335 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:15.335 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-03-11T23:16:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_10 2024-03-11T23:16:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_10 2024-03-11T23:16:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_10 2024-03-11T23:16:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-03-11T23:16:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-03-11T23:16:15.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:15.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:15.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:15.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:15.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_11 2024-03-11T23:16:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2024-03-11T23:16:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_11 2024-03-11T23:16:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_11 2024-03-11T23:16:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_11 2024-03-11T23:16:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-03-11T23:16:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:15.502 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-03-11T23:16:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_11 deleted '' 2024-03-11T23:16:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_11 2024-03-11T23:16:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_11 2024-03-11T23:16:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_11 2024-03-11T23:16:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-03-11T23:16:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-03-11T23:16:15.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:15.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:15.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:15.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:15.645 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:15.646 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_12 2024-03-11T23:16:15.646 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2024-03-11T23:16:15.646 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:15.646 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:15.646 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_12 2024-03-11T23:16:15.646 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:15.646 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:15.647 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:15.647 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:15.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_12 2024-03-11T23:16:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_12 2024-03-11T23:16:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-03-11T23:16:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:15.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-03-11T23:16:15.678 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:15.678 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:15.678 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:15.780 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_12 deleted '' 2024-03-11T23:16:15.780 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:15.780 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:15.780 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_12 2024-03-11T23:16:15.780 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:15.780 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:15.780 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:15.780 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:15.780 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_12 2024-03-11T23:16:15.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:15.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:15.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_12 2024-03-11T23:16:15.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-03-11T23:16:15.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-03-11T23:16:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_13 2024-03-11T23:16:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2024-03-11T23:16:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_13 2024-03-11T23:16:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_13 2024-03-11T23:16:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_13 2024-03-11T23:16:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-03-11T23:16:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:15.849 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-03-11T23:16:15.852 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:15.852 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:15.852 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:15.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_13 deleted '' 2024-03-11T23:16:15.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:15.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:15.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_13 2024-03-11T23:16:15.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:15.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:15.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:15.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:15.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_13 2024-03-11T23:16:15.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:15.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:15.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_13 2024-03-11T23:16:15.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-03-11T23:16:15.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:15.985 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-03-11T23:16:15.988 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:15.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:15.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:15.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:15.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:15.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_14 2024-03-11T23:16:15.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2024-03-11T23:16:15.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:15.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:15.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_14 2024-03-11T23:16:15.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:15.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:15.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:15.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_14 2024-03-11T23:16:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_14 2024-03-11T23:16:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-03-11T23:16:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:16.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-03-11T23:16:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_14 deleted '' 2024-03-11T23:16:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_14 2024-03-11T23:16:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_14 2024-03-11T23:16:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_14 2024-03-11T23:16:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-03-11T23:16:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:16.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-03-11T23:16:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_15 2024-03-11T23:16:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2024-03-11T23:16:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:16.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:16.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_15 2024-03-11T23:16:16.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:16.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:16.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:16.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:16.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_15 2024-03-11T23:16:16.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:16.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:16.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_15 2024-03-11T23:16:16.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-03-11T23:16:16.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:16.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-03-11T23:16:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:16.194 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:16.194 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:16.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_15 deleted '' 2024-03-11T23:16:16.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:16.295 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:16.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_15 2024-03-11T23:16:16.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:16.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:16.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:16.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:16.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_15 2024-03-11T23:16:16.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:16.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:16.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_15 2024-03-11T23:16:16.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-03-11T23:16:16.298 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:16.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-03-11T23:16:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_16 2024-03-11T23:16:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2024-03-11T23:16:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_16 2024-03-11T23:16:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_16 2024-03-11T23:16:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_16 2024-03-11T23:16:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-03-11T23:16:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:16.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-03-11T23:16:16.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:16.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:16.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_16 deleted '' 2024-03-11T23:16:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_16 2024-03-11T23:16:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_16 2024-03-11T23:16:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:16.470 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_16 2024-03-11T23:16:16.470 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-03-11T23:16:16.470 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:16.500 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-03-11T23:16:16.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:16.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:16.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_17 2024-03-11T23:16:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2024-03-11T23:16:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_17 2024-03-11T23:16:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_17 2024-03-11T23:16:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:16.506 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_17 2024-03-11T23:16:16.506 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-03-11T23:16:16.506 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:16.535 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-03-11T23:16:16.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:16.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:16.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:16.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_17 deleted '' 2024-03-11T23:16:16.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:16.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:16.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_17 2024-03-11T23:16:16.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:16.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:16.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:16.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:16.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_17 2024-03-11T23:16:16.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:16.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:16.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_17 2024-03-11T23:16:16.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-03-11T23:16:16.641 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:16.672 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-03-11T23:16:16.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:16.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:16.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:16.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:16.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:16.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_18 2024-03-11T23:16:16.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2024-03-11T23:16:16.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:16.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:16.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_18 2024-03-11T23:16:16.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:16.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:16.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:16.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:16.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_18 2024-03-11T23:16:16.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:16.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:16.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_18 2024-03-11T23:16:16.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-03-11T23:16:16.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-03-11T23:16:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:16.812 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_18 deleted '' 2024-03-11T23:16:16.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:16.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:16.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_18 2024-03-11T23:16:16.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:16.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:16.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:16.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:16.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_18 2024-03-11T23:16:16.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:16.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:16.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_18 2024-03-11T23:16:16.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-03-11T23:16:16.814 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-03-11T23:16:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_19 2024-03-11T23:16:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2024-03-11T23:16:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_19 2024-03-11T23:16:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_19 2024-03-11T23:16:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_19 2024-03-11T23:16:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-03-11T23:16:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-03-11T23:16:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_19 deleted '' 2024-03-11T23:16:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_19 2024-03-11T23:16:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_19 2024-03-11T23:16:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_19 2024-03-11T23:16:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-03-11T23:16:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-03-11T23:16:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_20 2024-03-11T23:16:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2024-03-11T23:16:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_20 2024-03-11T23:16:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_20 2024-03-11T23:16:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_20 2024-03-11T23:16:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-03-11T23:16:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:17.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-03-11T23:16:17.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:17.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:17.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:17.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_20 deleted '' 2024-03-11T23:16:17.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:17.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:17.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_20 2024-03-11T23:16:17.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_20 2024-03-11T23:16:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_20 2024-03-11T23:16:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-03-11T23:16:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:17.194 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-03-11T23:16:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_21 2024-03-11T23:16:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2024-03-11T23:16:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_21 2024-03-11T23:16:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_21 2024-03-11T23:16:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_21 2024-03-11T23:16:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-03-11T23:16:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:17.228 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-03-11T23:16:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:17.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_21 deleted '' 2024-03-11T23:16:17.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:17.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:17.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_21 2024-03-11T23:16:17.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_21 2024-03-11T23:16:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_21 2024-03-11T23:16:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-03-11T23:16:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:17.364 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-03-11T23:16:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_22 2024-03-11T23:16:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2024-03-11T23:16:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_22 2024-03-11T23:16:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_22 2024-03-11T23:16:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_22 2024-03-11T23:16:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-03-11T23:16:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-03-11T23:16:17.403 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:17.403 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:17.403 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_22 deleted '' 2024-03-11T23:16:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_22 2024-03-11T23:16:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_22 2024-03-11T23:16:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_22 2024-03-11T23:16:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-03-11T23:16:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-03-11T23:16:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_23 2024-03-11T23:16:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2024-03-11T23:16:17.544 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:17.544 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:17.544 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_23 2024-03-11T23:16:17.544 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:17.544 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:17.544 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:17.544 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:17.544 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_23 2024-03-11T23:16:17.545 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:17.545 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:17.545 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_23 2024-03-11T23:16:17.545 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-03-11T23:16:17.545 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-03-11T23:16:17.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:17.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:17.577 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:17.679 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_23 deleted '' 2024-03-11T23:16:17.679 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:17.679 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:17.679 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_23 2024-03-11T23:16:17.679 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:17.679 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_23 2024-03-11T23:16:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_23 2024-03-11T23:16:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-03-11T23:16:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-03-11T23:16:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_24 2024-03-11T23:16:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2024-03-11T23:16:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_24 2024-03-11T23:16:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_24 2024-03-11T23:16:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_24 2024-03-11T23:16:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-03-11T23:16:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-03-11T23:16:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_24 deleted '' 2024-03-11T23:16:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_24 2024-03-11T23:16:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_24 2024-03-11T23:16:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_24 2024-03-11T23:16:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-03-11T23:16:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-03-11T23:16:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:17.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:17.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:17.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_25 2024-03-11T23:16:17.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2024-03-11T23:16:17.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:17.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:17.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_25 2024-03-11T23:16:17.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:17.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:17.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:17.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:17.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_25 2024-03-11T23:16:17.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:17.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:17.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_25 2024-03-11T23:16:17.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-03-11T23:16:17.893 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-03-11T23:16:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_25 deleted '' 2024-03-11T23:16:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_25 2024-03-11T23:16:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_25 2024-03-11T23:16:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:18.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:18.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_25 2024-03-11T23:16:18.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-03-11T23:16:18.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:18.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-03-11T23:16:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_26 2024-03-11T23:16:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2024-03-11T23:16:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_26 2024-03-11T23:16:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_26 2024-03-11T23:16:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_26 2024-03-11T23:16:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-03-11T23:16:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:18.099 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-03-11T23:16:18.102 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:18.102 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:18.102 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_26 deleted '' 2024-03-11T23:16:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_26 2024-03-11T23:16:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_26 2024-03-11T23:16:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_26 2024-03-11T23:16:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-03-11T23:16:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:18.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-03-11T23:16:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_27 2024-03-11T23:16:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2024-03-11T23:16:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_27 2024-03-11T23:16:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_27 2024-03-11T23:16:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_27 2024-03-11T23:16:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-03-11T23:16:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:18.271 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-03-11T23:16:18.274 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:18.274 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:18.274 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:18.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_27 deleted '' 2024-03-11T23:16:18.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:18.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:18.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_27 2024-03-11T23:16:18.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:18.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:18.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:18.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:18.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_27 2024-03-11T23:16:18.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:18.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:18.377 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_27 2024-03-11T23:16:18.377 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-03-11T23:16:18.377 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-03-11T23:16:18.413 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:18.413 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:18.413 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:18.413 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:18.413 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_28 2024-03-11T23:16:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2024-03-11T23:16:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_28 2024-03-11T23:16:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_28 2024-03-11T23:16:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_28 2024-03-11T23:16:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-03-11T23:16:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:18.444 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-03-11T23:16:18.447 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:18.447 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:18.447 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:18.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_28 deleted '' 2024-03-11T23:16:18.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:18.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:18.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_28 2024-03-11T23:16:18.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:18.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:18.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:18.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:18.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_28 2024-03-11T23:16:18.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:18.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:18.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_28 2024-03-11T23:16:18.550 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-03-11T23:16:18.550 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-03-11T23:16:18.586 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:18.586 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:18.586 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:18.586 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:18.586 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:18.586 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_29 2024-03-11T23:16:18.586 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2024-03-11T23:16:18.586 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:18.587 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:18.587 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_29 2024-03-11T23:16:18.587 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:18.587 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:18.587 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:18.587 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:18.587 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_29 2024-03-11T23:16:18.587 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:18.587 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:18.587 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_29 2024-03-11T23:16:18.587 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-03-11T23:16:18.587 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-03-11T23:16:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_29 deleted '' 2024-03-11T23:16:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_29 2024-03-11T23:16:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_29 2024-03-11T23:16:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_29 2024-03-11T23:16:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-03-11T23:16:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-03-11T23:16:18.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_30 2024-03-11T23:16:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2024-03-11T23:16:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_30 2024-03-11T23:16:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_30 2024-03-11T23:16:18.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:18.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:18.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_30 2024-03-11T23:16:18.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-03-11T23:16:18.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:18.789 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-03-11T23:16:18.792 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:18.792 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:18.792 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:18.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_30 deleted '' 2024-03-11T23:16:18.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:18.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:18.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_30 2024-03-11T23:16:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_30 2024-03-11T23:16:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_30 2024-03-11T23:16:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-03-11T23:16:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-03-11T23:16:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_31 2024-03-11T23:16:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2024-03-11T23:16:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_31 2024-03-11T23:16:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_31 2024-03-11T23:16:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_31 2024-03-11T23:16:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-03-11T23:16:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-03-11T23:16:18.966 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:18.966 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:18.966 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:19.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_31 deleted '' 2024-03-11T23:16:19.068 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:19.068 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:19.068 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_31 2024-03-11T23:16:19.068 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:19.068 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:19.068 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:19.068 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:19.068 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_31 2024-03-11T23:16:19.068 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:19.068 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:19.068 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_31 2024-03-11T23:16:19.068 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-03-11T23:16:19.068 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-03-11T23:16:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_32 2024-03-11T23:16:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2024-03-11T23:16:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_32 2024-03-11T23:16:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_32 2024-03-11T23:16:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_32 2024-03-11T23:16:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-03-11T23:16:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-03-11T23:16:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_32 deleted '' 2024-03-11T23:16:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_32 2024-03-11T23:16:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_32 2024-03-11T23:16:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_32 2024-03-11T23:16:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-03-11T23:16:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-03-11T23:16:19.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:19.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:19.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:19.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:19.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_33 2024-03-11T23:16:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2024-03-11T23:16:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_33 2024-03-11T23:16:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_33 2024-03-11T23:16:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_33 2024-03-11T23:16:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-03-11T23:16:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-03-11T23:16:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_33 deleted '' 2024-03-11T23:16:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_33 2024-03-11T23:16:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_33 2024-03-11T23:16:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_33 2024-03-11T23:16:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-03-11T23:16:19.413 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-03-11T23:16:19.447 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:19.447 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:19.447 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_34 2024-03-11T23:16:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2024-03-11T23:16:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_34 2024-03-11T23:16:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_34 2024-03-11T23:16:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_34 2024-03-11T23:16:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-03-11T23:16:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-03-11T23:16:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_34 deleted '' 2024-03-11T23:16:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_34 2024-03-11T23:16:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_34 2024-03-11T23:16:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_34 2024-03-11T23:16:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-03-11T23:16:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:19.617 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-03-11T23:16:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_35 2024-03-11T23:16:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2024-03-11T23:16:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_35 2024-03-11T23:16:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:19.622 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:19.622 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:19.622 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_35 2024-03-11T23:16:19.622 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:19.622 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:19.622 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_35 2024-03-11T23:16:19.622 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-03-11T23:16:19.622 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:19.652 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-03-11T23:16:19.655 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:19.655 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:19.655 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:19.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_35 deleted '' 2024-03-11T23:16:19.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:19.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:19.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_35 2024-03-11T23:16:19.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:19.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:19.758 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:19.758 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:19.758 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_35 2024-03-11T23:16:19.758 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:19.758 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:19.758 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_35 2024-03-11T23:16:19.758 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-03-11T23:16:19.758 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-03-11T23:16:19.793 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:19.793 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:19.793 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:19.793 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:19.793 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:19.793 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_36 2024-03-11T23:16:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2024-03-11T23:16:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_36 2024-03-11T23:16:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_36 2024-03-11T23:16:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_36 2024-03-11T23:16:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-03-11T23:16:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:19.824 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-03-11T23:16:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:19.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_36 deleted '' 2024-03-11T23:16:19.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:19.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:19.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_36 2024-03-11T23:16:19.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:19.929 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_36 2024-03-11T23:16:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_36 2024-03-11T23:16:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-03-11T23:16:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-03-11T23:16:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_37 2024-03-11T23:16:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2024-03-11T23:16:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_37 2024-03-11T23:16:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_37 2024-03-11T23:16:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_37 2024-03-11T23:16:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-03-11T23:16:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-03-11T23:16:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_37 deleted '' 2024-03-11T23:16:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_37 2024-03-11T23:16:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_37 2024-03-11T23:16:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_37 2024-03-11T23:16:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-03-11T23:16:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-03-11T23:16:20.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:20.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:20.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:20.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:20.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:20.339 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_38 2024-03-11T23:16:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2024-03-11T23:16:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_38 2024-03-11T23:16:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_38 2024-03-11T23:16:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_38 2024-03-11T23:16:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-03-11T23:16:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-03-11T23:16:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:20.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_38 deleted '' 2024-03-11T23:16:20.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:20.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:20.475 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_38 2024-03-11T23:16:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_38 2024-03-11T23:16:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_38 2024-03-11T23:16:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-03-11T23:16:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-03-11T23:16:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_39 2024-03-11T23:16:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2024-03-11T23:16:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_39 2024-03-11T23:16:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_39 2024-03-11T23:16:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_39 2024-03-11T23:16:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-03-11T23:16:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:20.543 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-03-11T23:16:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_39 deleted '' 2024-03-11T23:16:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_39 2024-03-11T23:16:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_39 2024-03-11T23:16:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_39 2024-03-11T23:16:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-03-11T23:16:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-03-11T23:16:20.684 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:20.684 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:20.684 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:20.684 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:20.684 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:20.685 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_40 2024-03-11T23:16:20.685 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2024-03-11T23:16:20.685 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:20.685 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:20.685 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_40 2024-03-11T23:16:20.685 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:20.685 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:20.685 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:20.685 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:20.685 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_40 2024-03-11T23:16:20.685 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:20.685 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:20.685 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_40 2024-03-11T23:16:20.685 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-03-11T23:16:20.686 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-03-11T23:16:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_40 deleted '' 2024-03-11T23:16:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_40 2024-03-11T23:16:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_40 2024-03-11T23:16:20.821 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:20.821 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:20.821 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_40 2024-03-11T23:16:20.821 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-03-11T23:16:20.821 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-03-11T23:16:20.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:20.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:20.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:20.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:20.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_41 2024-03-11T23:16:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2024-03-11T23:16:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_41 2024-03-11T23:16:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_41 2024-03-11T23:16:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_41 2024-03-11T23:16:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-03-11T23:16:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:20.902 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-03-11T23:16:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:21.006 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_41 deleted '' 2024-03-11T23:16:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_41 2024-03-11T23:16:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_41 2024-03-11T23:16:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_41 2024-03-11T23:16:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-03-11T23:16:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:21.040 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-03-11T23:16:21.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:21.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:21.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:21.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:21.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:21.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_42 2024-03-11T23:16:21.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2024-03-11T23:16:21.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:21.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_42 2024-03-11T23:16:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_42 2024-03-11T23:16:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_42 2024-03-11T23:16:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-03-11T23:16:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-03-11T23:16:21.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:21.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_42 deleted '' 2024-03-11T23:16:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_42 2024-03-11T23:16:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_42 2024-03-11T23:16:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_42 2024-03-11T23:16:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-03-11T23:16:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-03-11T23:16:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_43 2024-03-11T23:16:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2024-03-11T23:16:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_43 2024-03-11T23:16:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:21.217 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_43 2024-03-11T23:16:21.217 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:21.217 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:21.217 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_43 2024-03-11T23:16:21.217 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-03-11T23:16:21.217 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:21.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-03-11T23:16:21.251 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:21.251 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:21.251 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_43 deleted '' 2024-03-11T23:16:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_43 2024-03-11T23:16:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_43 2024-03-11T23:16:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_43 2024-03-11T23:16:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-03-11T23:16:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:21.385 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-03-11T23:16:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_44 2024-03-11T23:16:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2024-03-11T23:16:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_44 2024-03-11T23:16:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_44 2024-03-11T23:16:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_44 2024-03-11T23:16:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-03-11T23:16:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:21.420 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-03-11T23:16:21.425 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:21.425 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:21.425 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_44 deleted '' 2024-03-11T23:16:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_44 2024-03-11T23:16:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_44 2024-03-11T23:16:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_44 2024-03-11T23:16:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-03-11T23:16:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:21.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-03-11T23:16:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_45 2024-03-11T23:16:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2024-03-11T23:16:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_45 2024-03-11T23:16:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_45 2024-03-11T23:16:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_45 2024-03-11T23:16:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-03-11T23:16:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:21.593 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-03-11T23:16:21.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:21.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:21.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:21.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_45 deleted '' 2024-03-11T23:16:21.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:21.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:21.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_45 2024-03-11T23:16:21.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:21.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:21.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:21.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:21.698 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_45 2024-03-11T23:16:21.699 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:21.699 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:21.699 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_45 2024-03-11T23:16:21.699 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-03-11T23:16:21.699 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-03-11T23:16:21.736 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:21.736 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:21.736 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:21.736 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:21.736 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_46 2024-03-11T23:16:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2024-03-11T23:16:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_46 2024-03-11T23:16:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_46 2024-03-11T23:16:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_46 2024-03-11T23:16:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-03-11T23:16:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-03-11T23:16:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_46 deleted '' 2024-03-11T23:16:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_46 2024-03-11T23:16:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_46 2024-03-11T23:16:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_46 2024-03-11T23:16:21.876 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-03-11T23:16:21.876 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-03-11T23:16:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_47 2024-03-11T23:16:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2024-03-11T23:16:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_47 2024-03-11T23:16:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_47 2024-03-11T23:16:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_47 2024-03-11T23:16:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-03-11T23:16:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:21.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-03-11T23:16:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:22.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_47 deleted '' 2024-03-11T23:16:22.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_47 2024-03-11T23:16:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_47 2024-03-11T23:16:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_47 2024-03-11T23:16:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-03-11T23:16:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:22.086 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-03-11T23:16:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_48 2024-03-11T23:16:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2024-03-11T23:16:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_48 2024-03-11T23:16:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_48 2024-03-11T23:16:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_48 2024-03-11T23:16:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-03-11T23:16:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-03-11T23:16:22.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:22.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:22.124 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_48 deleted '' 2024-03-11T23:16:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_48 2024-03-11T23:16:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_48 2024-03-11T23:16:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_48 2024-03-11T23:16:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-03-11T23:16:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:22.259 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-03-11T23:16:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_49 2024-03-11T23:16:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2024-03-11T23:16:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_49 2024-03-11T23:16:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_49 2024-03-11T23:16:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_49 2024-03-11T23:16:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-03-11T23:16:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:22.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-03-11T23:16:22.296 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:22.296 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:22.296 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:22.398 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_49 deleted '' 2024-03-11T23:16:22.398 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:22.398 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:22.398 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_49 2024-03-11T23:16:22.398 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:22.398 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:22.398 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:22.398 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:22.398 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_49 2024-03-11T23:16:22.398 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:22.398 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:22.399 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_49 2024-03-11T23:16:22.399 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-03-11T23:16:22.399 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-03-11T23:16:22.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:22.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:22.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:22.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:22.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-03-11T23:16:22.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_50 2024-03-11T23:16:22.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2024-03-11T23:16:22.433 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:22.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:22.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_50 2024-03-11T23:16:22.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:22.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:22.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:22.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:22.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_50 2024-03-11T23:16:22.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:22.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:22.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_50 2024-03-11T23:16:22.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-03-11T23:16:22.434 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-03-11T23:16:22.466 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:22.467 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:22.467 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_50 deleted '' 2024-03-11T23:16:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_50 2024-03-11T23:16:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_50 2024-03-11T23:16:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_50 2024-03-11T23:16:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-03-11T23:16:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-03-11T23:16:22.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:22.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:22.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:22.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:22.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ testlog 'TEST: delete images during bootstrap' 2024-03-11T23:16:22.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ tee -a /tmp/tmp.7mIVT0l9FS/rbd-mirror.test.log 2024-03-11T23:16:22.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ date '+%F %T' 2024-03-11T23:16:22.605 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ echo 2024-03-11 23:16:22 TEST: delete images during bootstrap 2024-03-11T23:16:22.605 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:2024-03-11 23:16:22 TEST: delete images during bootstrap 2024-03-11T23:16:22.605 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set_pool_mirror_mode cluster1 mirror image 2024-03-11T23:16:22.605 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:22.605 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:22.605 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=image 2024-03-11T23:16:22.605 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2024-03-11T23:16:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd: mirroring is already configured for image mode 2024-03-11T23:16:22.638 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set_pool_mirror_mode cluster2 mirror image 2024-03-11T23:16:22.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:16:22.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:22.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=image 2024-03-11T23:16:22.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2024-03-11T23:16:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd: mirroring is already configured for image mode 2024-03-11T23:16:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ start_mirror cluster1 2024-03-11T23:16:22.674 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:22.674 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local instance 2024-03-11T23:16:22.674 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set_cluster_instance cluster1 cluster instance 2024-03-11T23:16:22.674 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local val=cluster1 2024-03-11T23:16:22.674 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster_var_name=cluster 2024-03-11T23:16:22.674 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local instance_var_name=instance 2024-03-11T23:16:22.674 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ cluster=cluster1 2024-03-11T23:16:22.674 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ instance=cluster1 2024-03-11T23:16:22.674 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' cluster1 = cluster1 ']' 2024-03-11T23:16:22.674 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ instance=0 2024-03-11T23:16:22.674 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ eval cluster=cluster1 2024-03-11T23:16:22.674 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cluster=cluster1 2024-03-11T23:16:22.674 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ eval instance=0 2024-03-11T23:16:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ instance=0 2024-03-11T23:16:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n 1 2024-03-11T23:16:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 2024-03-11T23:16:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=test 2024-03-11T23:16:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ seq 1 10 2024-03-11T23:16:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T23:16:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_1 2024-03-11T23:16:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_1 512M 2024-03-11T23:16:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:16:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:16:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:16:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T23:16:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:16:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:16:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 512M ']' 2024-03-11T23:16:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=512M 2024-03-11T23:16:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:16:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_1 2024-03-11T23:16:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_1 2024-03-11T23:16:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:16:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T23:16:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:16:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2024-03-11T23:16:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:16:23.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-03-11T23:16:23.410 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_1': 2024-03-11T23:16:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 512 MiB in 128 objects 2024-03-11T23:16:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:16:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:16:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 18ecc366cee 2024-03-11T23:16:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.18ecc366cee 2024-03-11T23:16:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:16:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:16:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:16:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:16:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:16:22 2024 2024-03-11T23:16:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:16:22 2024 2024-03-11T23:16:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:16:22 2024 2024-03-11T23:16:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:16:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:16:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 673377d0-9dc5-40c0-9ade-053a8daaa435 2024-03-11T23:16:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:16:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_1 2024-03-11T23:16:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:16:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T23:16:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:16:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=14 2024-03-11T23:16:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:16:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:16:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 4428800 bytes 2024-03-11T23:16:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 10569728 bytes 2024-03-11T23:16:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 69773312 bytes 2024-03-11T23:16:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 111946752 bytes 2024-03-11T23:16:24.195 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 130246656 bytes 2024-03-11T23:16:24.409 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 187395072 bytes 2024-03-11T23:16:24.534 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 205859840 bytes 2024-03-11T23:16:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 278882304 bytes 2024-03-11T23:16:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 322710528 bytes 2024-03-11T23:16:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 381681664 bytes 2024-03-11T23:16:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 449125376 bytes 2024-03-11T23:16:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 466209792 bytes 2024-03-11T23:16:25.651 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 485454848 bytes 2024-03-11T23:16:25.853 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 540480512 bytes 2024-03-11T23:16:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 553271296 bytes 2024-03-11T23:16:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 596076544 bytes 2024-03-11T23:16:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 597143552 bytes 2024-03-11T23:16:26.430 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 635021312 bytes 2024-03-11T23:16:26.674 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 704578560 bytes 2024-03-11T23:16:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 766078976 bytes 2024-03-11T23:16:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 829689856 bytes 2024-03-11T23:16:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 933442560 bytes 2024-03-11T23:16:27.769 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 964025344 bytes 2024-03-11T23:16:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 1007206400 bytes 2024-03-11T23:16:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1008471040 bytes 2024-03-11T23:16:28.203 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1036661760 bytes 2024-03-11T23:16:28.423 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1094621184 bytes 2024-03-11T23:16:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1112669184 bytes 2024-03-11T23:16:28.775 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1171976192 bytes 2024-03-11T23:16:29.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1227485184 bytes 2024-03-11T23:16:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1228099584 bytes 2024-03-11T23:16:29.305 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1279816704 bytes 2024-03-11T23:16:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1287800832 bytes 2024-03-11T23:16:29.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1310915584 bytes 2024-03-11T23:16:29.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1366669312 bytes 2024-03-11T23:16:29.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1406645248 bytes 2024-03-11T23:16:30.172 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1464495104 bytes 2024-03-11T23:16:30.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1509841920 bytes 2024-03-11T23:16:30.546 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1563778048 bytes 2024-03-11T23:16:30.726 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1600536576 bytes 2024-03-11T23:16:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1644212224 bytes 2024-03-11T23:16:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1686420480 bytes 2024-03-11T23:16:31.302 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1725581312 bytes 2024-03-11T23:16:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 1760070656 bytes 2024-03-11T23:16:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 1797141504 bytes 2024-03-11T23:16:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 1832178688 bytes 2024-03-11T23:16:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 1899388928 bytes 2024-03-11T23:16:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 1927069696 bytes 2024-03-11T23:16:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 1980016640 bytes 2024-03-11T23:16:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 1980650496 bytes 2024-03-11T23:16:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5100 IOs, 2026591232 bytes 2024-03-11T23:16:32.865 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5200 IOs, 2081753088 bytes 2024-03-11T23:16:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5300 IOs, 2156819456 bytes 2024-03-11T23:16:33.239 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5400 IOs, 2157416448 bytes 2024-03-11T23:16:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5500 IOs, 2217670656 bytes 2024-03-11T23:16:33.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5600 IOs, 2250470400 bytes 2024-03-11T23:16:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5700 IOs, 2289062912 bytes 2024-03-11T23:16:33.972 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5800 IOs, 2331735040 bytes 2024-03-11T23:16:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5900 IOs, 2398838784 bytes 2024-03-11T23:16:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6000 IOs, 2496217088 bytes 2024-03-11T23:16:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6100 IOs, 2527837184 bytes 2024-03-11T23:16:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6200 IOs, 2556922880 bytes 2024-03-11T23:16:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6300 IOs, 2598890496 bytes 2024-03-11T23:16:35.178 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6400 IOs, 2647096320 bytes 2024-03-11T23:16:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6500 IOs, 2675889152 bytes 2024-03-11T23:16:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6600 IOs, 2744651776 bytes 2024-03-11T23:16:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6700 IOs, 2775939072 bytes 2024-03-11T23:16:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6800 IOs, 2787532800 bytes 2024-03-11T23:16:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6900 IOs, 2865606656 bytes 2024-03-11T23:16:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7000 IOs, 2928790528 bytes 2024-03-11T23:16:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7100 IOs, 2989012992 bytes 2024-03-11T23:16:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7200 IOs, 2989623296 bytes 2024-03-11T23:16:36.837 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7300 IOs, 3026254848 bytes 2024-03-11T23:16:37.068 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7400 IOs, 3095732224 bytes 2024-03-11T23:16:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7500 IOs, 3107110912 bytes 2024-03-11T23:16:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:16:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:16:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:16:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2024-03-11T23:16:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T23:16:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=present 2024-03-11T23:16:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_1 2024-03-11T23:16:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_1 2024-03-11T23:16:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-03-11T23:16:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id=12b0b9e1ffda 2024-03-11T23:16:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:37.463 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_1 present 12b0b9e1ffda 2024-03-11T23:16:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T23:16:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=present 2024-03-11T23:16:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id=12b0b9e1ffda 2024-03-11T23:16:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_1 2024-03-11T23:16:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_1 2024-03-11T23:16:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-03-11T23:16:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:37.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id=12b0b9e1ffda 2024-03-11T23:16:37.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n 12b0b9e1ffda 2024-03-11T23:16:37.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -z 12b0b9e1ffda -o 12b0b9e1ffda = 12b0b9e1ffda 2024-03-11T23:16:37.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=present 2024-03-11T23:16:37.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test present = present 2024-03-11T23:16:37.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:37.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ disable_mirror cluster2 mirror image_1 2024-03-11T23:16:37.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:16:37.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:37.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T23:16:37.608 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2024-03-11T23:16:38.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi132.stderr:2024-03-11T23:16:38.440+0000 7f84b4efc640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55ba0f426000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=185 2024-03-11T23:16:38.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi132.stderr:2024-03-11T23:16:38.440+0000 7f84b4efc640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55ba0f426000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=185, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=baf1db9c-11db-489a-a01f-16a50bade1d5, primary_snap_id=b9, last_copied_object_number=128, snap_seqs={185=18446744073709551614}] 2024-03-11T23:16:38.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi132.stderr:2024-03-11T23:16:38.787+0000 7fac0f35f640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5629d2e249c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:16:38.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi132.stderr:2024-03-11T23:16:38.787+0000 7fac15b6c640 -1 rbd::mirror::ImageReplayer: 0x5629d3376f00 [3/673377d0-9dc5-40c0-9ade-053a8daaa435] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:16:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring disabled 2024-03-11T23:16:43.456 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-03-11T23:16:43.456 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:43.456 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:43.456 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T23:16:43.456 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:16:43.456 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:43.456 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:16:43.456 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:43.456 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_1 2024-03-11T23:16:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_1 2024-03-11T23:16:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-03-11T23:16:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-03-11T23:16:43.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:16:43.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:16:43.496 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:16:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-03-11T23:16:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:16:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:43.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T23:16:43.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:16:43.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:16:43.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:16:43.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:16:43.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_1 2024-03-11T23:16:43.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:16:43.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:16:43.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_1 2024-03-11T23:16:43.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-03-11T23:16:43.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:16:43.631 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-03-11T23:16:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:16:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:16:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:16:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:16:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ purge_snapshots cluster2 mirror image_1 2024-03-11T23:16:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:16:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T23:16:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2024-03-11T23:16:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_1 2024-03-11T23:16:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:16:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T23:16:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:16:43.678 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:16:43.678 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_1 2024-03-11T23:16:43.678 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:16:43.678 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:43.678 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T23:16:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-03-11T23:16:43.720 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-03-11T23:16:43.762 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 0% complete...failed. 2024-03-11T23:16:43.763 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error: image still has watchers 2024-03-11T23:16:43.763 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-03-11T23:16:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:16:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:16:44.767 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_1 2024-03-11T23:16:44.767 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:16:44.767 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:44.767 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T23:16:44.767 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-03-11T23:16:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-03-11T23:16:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 0% complete...failed. 2024-03-11T23:16:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error: image still has watchers 2024-03-11T23:16:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-03-11T23:16:44.853 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:16:44.853 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 2 2024-03-11T23:16:44.953 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9593569390701288) > weight (0.25) 2024-03-11T23:16:44.953 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-03-11T23:16:44.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-03-11T23:16:44.953 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-03-11T23:16:44.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr:2024-03-11T23:16:44.951+0000 7efd32b88640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 68403) UID: 1000 2024-03-11T23:16:46.854 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_1 2024-03-11T23:16:46.854 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:16:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T23:16:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-03-11T23:16:46.905 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-03-11T23:16:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 0% complete...failed. 2024-03-11T23:16:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error: image still has watchers 2024-03-11T23:16:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-03-11T23:16:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:16:46.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 4 2024-03-11T23:16:46.954 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-03-11T23:16:46.954 INFO:teuthology.orchestra.run:waiting for 600 2024-03-11T23:16:46.954 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-03-11T23:16:46.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-03-11T23:16:46.955 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-03-11T23:16:46.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-03-11T23:16:46.957 INFO:tasks.rbd_mirror_thrash:waiting for 104 secs before thrashing 2024-03-11T23:16:50.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_1 2024-03-11T23:16:50.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:16:50.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T23:16:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-03-11T23:16:51.003 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-03-11T23:16:51.048 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 0% complete...failed. 2024-03-11T23:16:51.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error: image still has watchers 2024-03-11T23:16:51.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-03-11T23:16:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:16:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 8 2024-03-11T23:16:59.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_1 2024-03-11T23:16:59.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:16:59.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:16:59.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T23:16:59.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-03-11T23:16:59.103 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-03-11T23:16:59.148 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 0% complete...failed. 2024-03-11T23:16:59.148 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error: image still has watchers 2024-03-11T23:16:59.148 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-03-11T23:16:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:16:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 16 2024-03-11T23:17:12.933 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5147119704431523) > weight (0.25) 2024-03-11T23:17:12.933 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5304521027956995) > weight (0.25) 2024-03-11T23:17:12.934 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6659890765155835) > weight (0.25) 2024-03-11T23:17:12.934 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-03-11T23:17:12.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-03-11T23:17:12.934 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-03-11T23:17:12.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi132.stderr:2024-03-11T23:17:12.932+0000 7fac16b6e640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 42995) UID: 1000 2024-03-11T23:17:15.152 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_1 2024-03-11T23:17:15.152 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:17:15.152 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:17:15.152 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_1 2024-03-11T23:17:15.152 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-03-11T23:17:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-03-11T23:17:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-03-11T23:17:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:17:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T23:17:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_2 2024-03-11T23:17:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_2 512M 2024-03-11T23:17:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:17:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:17:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:17:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:17:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T23:17:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:17:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:17:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 512M ']' 2024-03-11T23:17:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=512M 2024-03-11T23:17:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:17:15.444 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_2 2024-03-11T23:17:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_2 2024-03-11T23:17:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:17:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:17:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T23:17:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:17:15.488 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2024-03-11T23:17:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:17:16.058 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-03-11T23:17:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_2': 2024-03-11T23:17:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 512 MiB in 128 objects 2024-03-11T23:17:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:17:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:17:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 1927872df62c 2024-03-11T23:17:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.1927872df62c 2024-03-11T23:17:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:17:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:17:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:17:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:17:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:17:15 2024 2024-03-11T23:17:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:17:15 2024 2024-03-11T23:17:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:17:15 2024 2024-03-11T23:17:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:17:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:17:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 15b8affb-f52a-4327-89b7-270108112c18 2024-03-11T23:17:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:17:16.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_2 2024-03-11T23:17:16.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:17:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:17:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T23:17:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:17:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=8 2024-03-11T23:17:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:17:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:17:16.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 20020224 bytes 2024-03-11T23:17:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 52184064 bytes 2024-03-11T23:17:16.564 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 79484928 bytes 2024-03-11T23:17:16.754 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 135334912 bytes 2024-03-11T23:17:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 170569728 bytes 2024-03-11T23:17:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 249364480 bytes 2024-03-11T23:17:17.359 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 290919424 bytes 2024-03-11T23:17:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 317639680 bytes 2024-03-11T23:17:17.670 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 356947968 bytes 2024-03-11T23:17:17.822 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 380976128 bytes 2024-03-11T23:17:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 453899264 bytes 2024-03-11T23:17:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 517828608 bytes 2024-03-11T23:17:18.416 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 539975680 bytes 2024-03-11T23:17:18.562 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 566496256 bytes 2024-03-11T23:17:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 587851776 bytes 2024-03-11T23:17:19.006 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 682296320 bytes 2024-03-11T23:17:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 711256064 bytes 2024-03-11T23:17:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 753007616 bytes 2024-03-11T23:17:19.506 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr:2024-03-11T23:17:19.503+0000 7fc263e49640 -1 bdev(0x55a8730e0e00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2024-03-11T23:17:19.507 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1979-gb117a47a/rpm/el9/BUILD/ceph-19.0.0-1979-gb117a47a/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fc263e49640 time 2024-03-11T23:17:19.504453+0000 2024-03-11T23:17:19.507 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1979-gb117a47a/rpm/el9/BUILD/ceph-19.0.0-1979-gb117a47a/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2024-03-11T23:17:19.508 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: ceph version 19.0.0-1979-gb117a47a (b117a47aa0ad81d207fdd9ba852068f2f262fea8) squid (dev) 2024-03-11T23:17:19.508 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x55a8702af4a3] 2024-03-11T23:17:19.508 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 2: (KernelDevice::_aio_thread()+0x11e0) [0x55a870c80550] 2024-03-11T23:17:19.508 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 3: ceph-osd(+0xdc66c1) [0x55a870c806c1] 2024-03-11T23:17:19.508 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 4: /lib64/libc.so.6(+0x9f802) [0x7fc27149f802] 2024-03-11T23:17:19.508 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 5: /lib64/libc.so.6(+0x3f450) [0x7fc27143f450] 2024-03-11T23:17:19.508 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr:2024-03-11T23:17:19.504+0000 7fc263e49640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1979-gb117a47a/rpm/el9/BUILD/ceph-19.0.0-1979-gb117a47a/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fc263e49640 time 2024-03-11T23:17:19.504453+0000 2024-03-11T23:17:19.508 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1979-gb117a47a/rpm/el9/BUILD/ceph-19.0.0-1979-gb117a47a/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2024-03-11T23:17:19.508 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 2024-03-11T23:17:19.508 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: ceph version 19.0.0-1979-gb117a47a (b117a47aa0ad81d207fdd9ba852068f2f262fea8) squid (dev) 2024-03-11T23:17:19.508 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x55a8702af4a3] 2024-03-11T23:17:19.508 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 2: (KernelDevice::_aio_thread()+0x11e0) [0x55a870c80550] 2024-03-11T23:17:19.508 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 3: ceph-osd(+0xdc66c1) [0x55a870c806c1] 2024-03-11T23:17:19.509 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 4: /lib64/libc.so.6(+0x9f802) [0x7fc27149f802] 2024-03-11T23:17:19.509 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 5: /lib64/libc.so.6(+0x3f450) [0x7fc27143f450] 2024-03-11T23:17:19.509 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 2024-03-11T23:17:19.509 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr:*** Caught signal (Aborted) ** 2024-03-11T23:17:19.509 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: in thread 7fc263e49640 thread_name:bstore_aio 2024-03-11T23:17:19.509 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: ceph version 19.0.0-1979-gb117a47a (b117a47aa0ad81d207fdd9ba852068f2f262fea8) squid (dev) 2024-03-11T23:17:19.509 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7fc271454db0] 2024-03-11T23:17:19.510 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7fc2714a154c] 2024-03-11T23:17:19.510 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 3: raise() 2024-03-11T23:17:19.510 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 4: abort() 2024-03-11T23:17:19.510 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x55a8702af564] 2024-03-11T23:17:19.510 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 6: (KernelDevice::_aio_thread()+0x11e0) [0x55a870c80550] 2024-03-11T23:17:19.510 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 7: ceph-osd(+0xdc66c1) [0x55a870c806c1] 2024-03-11T23:17:19.510 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7fc27149f802] 2024-03-11T23:17:19.510 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7fc27143f450] 2024-03-11T23:17:19.510 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr:2024-03-11T23:17:19.505+0000 7fc263e49640 -1 *** Caught signal (Aborted) ** 2024-03-11T23:17:19.510 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: in thread 7fc263e49640 thread_name:bstore_aio 2024-03-11T23:17:19.510 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 2024-03-11T23:17:19.510 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: ceph version 19.0.0-1979-gb117a47a (b117a47aa0ad81d207fdd9ba852068f2f262fea8) squid (dev) 2024-03-11T23:17:19.510 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7fc271454db0] 2024-03-11T23:17:19.511 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7fc2714a154c] 2024-03-11T23:17:19.511 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 3: raise() 2024-03-11T23:17:19.511 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 4: abort() 2024-03-11T23:17:19.511 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x55a8702af564] 2024-03-11T23:17:19.511 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 6: (KernelDevice::_aio_thread()+0x11e0) [0x55a870c80550] 2024-03-11T23:17:19.511 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 7: ceph-osd(+0xdc66c1) [0x55a870c806c1] 2024-03-11T23:17:19.511 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7fc27149f802] 2024-03-11T23:17:19.511 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7fc27143f450] 2024-03-11T23:17:19.511 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-11T23:17:19.511 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 2024-03-11T23:17:19.522 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: -23> 2024-03-11T23:17:19.503+0000 7fc263e49640 -1 bdev(0x55a8730e0e00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2024-03-11T23:17:19.523 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: -19> 2024-03-11T23:17:19.504+0000 7fc263e49640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1979-gb117a47a/rpm/el9/BUILD/ceph-19.0.0-1979-gb117a47a/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fc263e49640 time 2024-03-11T23:17:19.504453+0000 2024-03-11T23:17:19.523 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1979-gb117a47a/rpm/el9/BUILD/ceph-19.0.0-1979-gb117a47a/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2024-03-11T23:17:19.523 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 2024-03-11T23:17:19.523 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: ceph version 19.0.0-1979-gb117a47a (b117a47aa0ad81d207fdd9ba852068f2f262fea8) squid (dev) 2024-03-11T23:17:19.523 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x55a8702af4a3] 2024-03-11T23:17:19.523 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 2: (KernelDevice::_aio_thread()+0x11e0) [0x55a870c80550] 2024-03-11T23:17:19.523 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 3: ceph-osd(+0xdc66c1) [0x55a870c806c1] 2024-03-11T23:17:19.523 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 4: /lib64/libc.so.6(+0x9f802) [0x7fc27149f802] 2024-03-11T23:17:19.523 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 5: /lib64/libc.so.6(+0x3f450) [0x7fc27143f450] 2024-03-11T23:17:19.523 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 2024-03-11T23:17:19.523 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 0> 2024-03-11T23:17:19.505+0000 7fc263e49640 -1 *** Caught signal (Aborted) ** 2024-03-11T23:17:19.523 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: in thread 7fc263e49640 thread_name:bstore_aio 2024-03-11T23:17:19.524 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 2024-03-11T23:17:19.524 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: ceph version 19.0.0-1979-gb117a47a (b117a47aa0ad81d207fdd9ba852068f2f262fea8) squid (dev) 2024-03-11T23:17:19.524 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7fc271454db0] 2024-03-11T23:17:19.524 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7fc2714a154c] 2024-03-11T23:17:19.524 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 3: raise() 2024-03-11T23:17:19.524 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 4: abort() 2024-03-11T23:17:19.524 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x55a8702af564] 2024-03-11T23:17:19.524 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 6: (KernelDevice::_aio_thread()+0x11e0) [0x55a870c80550] 2024-03-11T23:17:19.524 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 7: ceph-osd(+0xdc66c1) [0x55a870c806c1] 2024-03-11T23:17:19.524 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7fc27149f802] 2024-03-11T23:17:19.524 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7fc27143f450] 2024-03-11T23:17:19.524 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-11T23:17:19.524 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 2024-03-11T23:17:19.532 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: -122> 2024-03-11T23:17:19.503+0000 7fc263e49640 -1 bdev(0x55a8730e0e00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2024-03-11T23:17:19.532 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: -118> 2024-03-11T23:17:19.504+0000 7fc263e49640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1979-gb117a47a/rpm/el9/BUILD/ceph-19.0.0-1979-gb117a47a/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fc263e49640 time 2024-03-11T23:17:19.504453+0000 2024-03-11T23:17:19.532 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1979-gb117a47a/rpm/el9/BUILD/ceph-19.0.0-1979-gb117a47a/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2024-03-11T23:17:19.532 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 2024-03-11T23:17:19.532 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: ceph version 19.0.0-1979-gb117a47a (b117a47aa0ad81d207fdd9ba852068f2f262fea8) squid (dev) 2024-03-11T23:17:19.532 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x55a8702af4a3] 2024-03-11T23:17:19.532 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 2: (KernelDevice::_aio_thread()+0x11e0) [0x55a870c80550] 2024-03-11T23:17:19.532 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 3: ceph-osd(+0xdc66c1) [0x55a870c806c1] 2024-03-11T23:17:19.532 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 4: /lib64/libc.so.6(+0x9f802) [0x7fc27149f802] 2024-03-11T23:17:19.532 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 5: /lib64/libc.so.6(+0x3f450) [0x7fc27143f450] 2024-03-11T23:17:19.532 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 2024-03-11T23:17:19.533 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: -99> 2024-03-11T23:17:19.505+0000 7fc263e49640 -1 *** Caught signal (Aborted) ** 2024-03-11T23:17:19.533 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: in thread 7fc263e49640 thread_name:bstore_aio 2024-03-11T23:17:19.533 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 2024-03-11T23:17:19.533 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: ceph version 19.0.0-1979-gb117a47a (b117a47aa0ad81d207fdd9ba852068f2f262fea8) squid (dev) 2024-03-11T23:17:19.533 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7fc271454db0] 2024-03-11T23:17:19.533 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7fc2714a154c] 2024-03-11T23:17:19.533 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 3: raise() 2024-03-11T23:17:19.533 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 4: abort() 2024-03-11T23:17:19.533 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x55a8702af564] 2024-03-11T23:17:19.533 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 6: (KernelDevice::_aio_thread()+0x11e0) [0x55a870c80550] 2024-03-11T23:17:19.533 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 7: ceph-osd(+0xdc66c1) [0x55a870c806c1] 2024-03-11T23:17:19.533 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7fc27149f802] 2024-03-11T23:17:19.533 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7fc27143f450] 2024-03-11T23:17:19.533 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-11T23:17:19.534 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr: 2024-03-11T23:17:19.612 INFO:tasks.ceph.cluster2.osd.0.smithi132.stderr:daemon-helper: command failed with exit status 5 2024-03-11T23:17:19.935 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-03-11T23:17:19.935 INFO:teuthology.orchestra.run:waiting for 600 2024-03-11T23:17:19.936 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-03-11T23:17:19.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-03-11T23:17:19.936 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-03-11T23:17:19.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-03-11T23:17:19.939 INFO:tasks.rbd_mirror_thrash:waiting for 73 secs before thrashing 2024-03-11T23:17:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 850132992 bytes 2024-03-11T23:17:21.309 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 858097664 bytes 2024-03-11T23:17:21.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 921047040 bytes 2024-03-11T23:17:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 933783552 bytes 2024-03-11T23:17:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 958227456 bytes 2024-03-11T23:17:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 1019786240 bytes 2024-03-11T23:17:22.019 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1095327744 bytes 2024-03-11T23:17:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1159512064 bytes 2024-03-11T23:17:22.327 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~0s 2024-03-11T23:17:22.375 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1231054848 bytes 2024-03-11T23:17:22.445 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1231686656 bytes 2024-03-11T23:17:22.540 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1274056704 bytes 2024-03-11T23:17:22.662 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1279448064 bytes 2024-03-11T23:17:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1335091200 bytes 2024-03-11T23:17:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1395795968 bytes 2024-03-11T23:17:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1429083136 bytes 2024-03-11T23:17:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1512701952 bytes 2024-03-11T23:17:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1561042944 bytes 2024-03-11T23:17:23.555 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1606864896 bytes 2024-03-11T23:17:23.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1657432064 bytes 2024-03-11T23:17:23.830 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1699109888 bytes 2024-03-11T23:17:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1752435712 bytes 2024-03-11T23:17:24.103 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:17:24.103 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:17:24.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:17:24.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:17:24.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2024-03-11T23:17:24.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:17:24.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:17:24.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T23:17:24.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=present 2024-03-11T23:17:24.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:17:24.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:17:24.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:17:24.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_2 2024-03-11T23:17:24.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:17:24.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:17:24.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_2 2024-03-11T23:17:24.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-03-11T23:17:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:17:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id=12b092c10640 2024-03-11T23:17:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:17:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:17:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_2 present 12b092c10640 2024-03-11T23:17:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:17:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:17:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T23:17:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=present 2024-03-11T23:17:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id=12b092c10640 2024-03-11T23:17:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:17:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:17:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_2 2024-03-11T23:17:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:17:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:17:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_2 2024-03-11T23:17:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-03-11T23:17:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:17:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id=12b092c10640 2024-03-11T23:17:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n 12b092c10640 2024-03-11T23:17:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -z 12b092c10640 -o 12b092c10640 = 12b092c10640 2024-03-11T23:17:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=present 2024-03-11T23:17:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test present = present 2024-03-11T23:17:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:17:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ disable_mirror cluster2 mirror image_2 2024-03-11T23:17:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:17:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:17:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T23:17:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2024-03-11T23:17:25.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi132.stderr:2024-03-11T23:17:25.132+0000 7f84b56fd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55ba0f3d8400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=186 2024-03-11T23:17:25.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi132.stderr:2024-03-11T23:17:25.132+0000 7f84b56fd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55ba0f3d8400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=186, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=baf1db9c-11db-489a-a01f-16a50bade1d5, primary_snap_id=ba, last_copied_object_number=128, snap_seqs={186=18446744073709551614}] 2024-03-11T23:17:27.732 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~5s 2024-03-11T23:17:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring disabled 2024-03-11T23:17:30.148 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-03-11T23:17:30.148 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:17:30.148 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:17:30.148 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T23:17:30.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:17:30.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:17:30.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:17:30.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:17:30.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_2 2024-03-11T23:17:30.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:17:30.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:17:30.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_2 2024-03-11T23:17:30.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-03-11T23:17:30.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:17:30.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi132.stderr:2024-03-11T23:17:30.166+0000 7f4eab017640 -1 rbd::mirror::ImageMap: 0x556661f890e0 handle_peer_ack_remove: failed to remove global_image_id=15b8affb-f52a-4327-89b7-270108112c18 2024-03-11T23:17:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-03-11T23:17:30.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:17:30.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:17:30.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:17:30.292 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-03-11T23:17:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:17:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:17:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T23:17:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:17:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:17:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:17:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:17:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_2 2024-03-11T23:17:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:17:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:17:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_2 2024-03-11T23:17:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-03-11T23:17:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:17:30.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-03-11T23:17:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:17:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:17:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:17:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:17:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ purge_snapshots cluster2 mirror image_2 2024-03-11T23:17:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:17:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:17:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T23:17:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2024-03-11T23:17:30.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_2 2024-03-11T23:17:30.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:17:30.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:17:30.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T23:17:30.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:17:30.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:17:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_2 2024-03-11T23:17:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:17:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:17:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T23:17:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-03-11T23:17:30.417 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-03-11T23:17:30.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 0% complete...failed. 2024-03-11T23:17:30.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error: image still has watchers 2024-03-11T23:17:30.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-03-11T23:17:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:17:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:17:31.462 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_2 2024-03-11T23:17:31.462 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:17:31.463 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:17:31.463 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T23:17:31.463 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-03-11T23:17:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-03-11T23:17:31.555 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 0% complete...failed. 2024-03-11T23:17:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error: image still has watchers 2024-03-11T23:17:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-03-11T23:17:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:17:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 2 2024-03-11T23:17:33.136 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~11s 2024-03-11T23:17:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_2 2024-03-11T23:17:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:17:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:17:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T23:17:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-03-11T23:17:33.611 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-03-11T23:17:33.656 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 0% complete...failed. 2024-03-11T23:17:33.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error: image still has watchers 2024-03-11T23:17:33.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-03-11T23:17:33.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:17:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 4 2024-03-11T23:17:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_2 2024-03-11T23:17:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:17:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:17:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T23:17:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-03-11T23:17:37.710 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-03-11T23:17:37.754 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 0% complete...failed. 2024-03-11T23:17:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error: image still has watchers 2024-03-11T23:17:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-03-11T23:17:37.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:17:37.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 8 2024-03-11T23:17:38.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~16s 2024-03-11T23:17:43.946 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~22s 2024-03-11T23:17:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_2 2024-03-11T23:17:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:17:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:17:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T23:17:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-03-11T23:17:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-03-11T23:17:45.852 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 0% complete...failed. 2024-03-11T23:17:45.852 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error: image still has watchers 2024-03-11T23:17:45.852 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-03-11T23:17:45.855 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:17:45.855 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 16 2024-03-11T23:17:49.351 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~27s 2024-03-11T23:17:54.755 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~32s 2024-03-11T23:18:00.159 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~38s 2024-03-11T23:18:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_2 2024-03-11T23:18:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:18:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:18:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_2 2024-03-11T23:18:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-03-11T23:18:01.909 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-03-11T23:18:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-03-11T23:18:02.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:18:02.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T23:18:02.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_3 2024-03-11T23:18:02.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_3 512M 2024-03-11T23:18:02.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:18:02.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:18:02.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:18:02.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:18:02.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T23:18:02.100 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:18:02.101 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:18:02.101 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 512M ']' 2024-03-11T23:18:02.101 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=512M 2024-03-11T23:18:02.101 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:18:02.101 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_3 2024-03-11T23:18:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_3 2024-03-11T23:18:02.146 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:18:02.146 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:18:02.146 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T23:18:02.146 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:18:02.146 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2024-03-11T23:18:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:18:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-03-11T23:18:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_3': 2024-03-11T23:18:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 512 MiB in 128 objects 2024-03-11T23:18:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:18:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:18:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 19641cb463a7 2024-03-11T23:18:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.19641cb463a7 2024-03-11T23:18:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:18:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:18:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:18:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:18:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:18:02 2024 2024-03-11T23:18:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:18:02 2024 2024-03-11T23:18:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:18:02 2024 2024-03-11T23:18:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:18:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:18:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 38dc9c32-cb20-400c-b098-8af4839492ab 2024-03-11T23:18:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:18:03.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_3 2024-03-11T23:18:03.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:18:03.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:18:03.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T23:18:03.237 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:18:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=5 2024-03-11T23:18:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:18:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:18:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 50003968 bytes 2024-03-11T23:18:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 113611776 bytes 2024-03-11T23:18:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 142050304 bytes 2024-03-11T23:18:03.881 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 231358464 bytes 2024-03-11T23:18:04.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 271828992 bytes 2024-03-11T23:18:04.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 291216384 bytes 2024-03-11T23:18:04.206 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 300788736 bytes 2024-03-11T23:18:04.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 358259712 bytes 2024-03-11T23:18:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 385894400 bytes 2024-03-11T23:18:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 407822336 bytes 2024-03-11T23:18:04.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 436431872 bytes 2024-03-11T23:18:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 477972480 bytes 2024-03-11T23:18:04.931 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 555401216 bytes 2024-03-11T23:18:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 602993664 bytes 2024-03-11T23:18:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 656588800 bytes 2024-03-11T23:18:05.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 673257472 bytes 2024-03-11T23:18:05.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 681447424 bytes 2024-03-11T23:18:05.526 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 742897664 bytes 2024-03-11T23:18:05.563 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~43s 2024-03-11T23:18:05.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 755992576 bytes 2024-03-11T23:18:05.789 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 833780736 bytes 2024-03-11T23:18:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 837329920 bytes 2024-03-11T23:18:05.988 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 864001024 bytes 2024-03-11T23:18:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 894430208 bytes 2024-03-11T23:18:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 929682432 bytes 2024-03-11T23:18:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 978890752 bytes 2024-03-11T23:18:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1031786496 bytes 2024-03-11T23:18:06.642 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1097455616 bytes 2024-03-11T23:18:06.821 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1160174592 bytes 2024-03-11T23:18:06.936 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1202711552 bytes 2024-03-11T23:18:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1255857152 bytes 2024-03-11T23:18:07.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1283147776 bytes 2024-03-11T23:18:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1311764480 bytes 2024-03-11T23:18:07.422 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1353021440 bytes 2024-03-11T23:18:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1383002112 bytes 2024-03-11T23:18:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1397554176 bytes 2024-03-11T23:18:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1427805184 bytes 2024-03-11T23:18:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1460794368 bytes 2024-03-11T23:18:07.958 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1481720832 bytes 2024-03-11T23:18:08.081 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1514127360 bytes 2024-03-11T23:18:08.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:18:08.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:18:08.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:18:08.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:18:08.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2024-03-11T23:18:08.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:18:08.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:18:08.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T23:18:08.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=present 2024-03-11T23:18:08.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:18:08.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:18:08.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:18:08.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_3 2024-03-11T23:18:08.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:18:08.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:18:08.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_3 2024-03-11T23:18:08.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-03-11T23:18:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:18:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id=12b0accce914 2024-03-11T23:18:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:18:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:18:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_3 present 12b0accce914 2024-03-11T23:18:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:18:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:18:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T23:18:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=present 2024-03-11T23:18:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id=12b0accce914 2024-03-11T23:18:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:18:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:18:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_3 2024-03-11T23:18:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:18:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:18:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_3 2024-03-11T23:18:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-03-11T23:18:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:18:08.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id=12b0accce914 2024-03-11T23:18:08.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n 12b0accce914 2024-03-11T23:18:08.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -z 12b0accce914 -o 12b0accce914 = 12b0accce914 2024-03-11T23:18:08.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=present 2024-03-11T23:18:08.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test present = present 2024-03-11T23:18:08.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:18:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ disable_mirror cluster2 mirror image_3 2024-03-11T23:18:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:18:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:18:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T23:18:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2024-03-11T23:18:09.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi132.stderr:2024-03-11T23:18:09.201+0000 7f84b4efc640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55ba0f53a400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=187 2024-03-11T23:18:09.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi132.stderr:2024-03-11T23:18:09.201+0000 7f84b4efc640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55ba0f53a400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=187, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=baf1db9c-11db-489a-a01f-16a50bade1d5, primary_snap_id=bb, last_copied_object_number=128, snap_seqs={187=18446744073709551614}] 2024-03-11T23:18:10.967 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~49s 2024-03-11T23:18:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring disabled 2024-03-11T23:18:14.215 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-03-11T23:18:14.215 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:18:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:18:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T23:18:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:18:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:18:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:18:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:18:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_3 2024-03-11T23:18:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:18:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:18:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_3 2024-03-11T23:18:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-03-11T23:18:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:18:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-03-11T23:18:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:18:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:18:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:18:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-03-11T23:18:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:18:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:18:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T23:18:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:18:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:18:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:18:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:18:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_3 2024-03-11T23:18:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:18:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:18:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_3 2024-03-11T23:18:14.361 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-03-11T23:18:14.361 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:18:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-03-11T23:18:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:18:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:18:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:18:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:18:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ purge_snapshots cluster2 mirror image_3 2024-03-11T23:18:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:18:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:18:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T23:18:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2024-03-11T23:18:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_3 2024-03-11T23:18:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:18:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:18:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T23:18:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:18:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:18:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_3 2024-03-11T23:18:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:18:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:18:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T23:18:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-03-11T23:18:14.489 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-03-11T23:18:14.530 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 0% complete...failed. 2024-03-11T23:18:14.530 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error: image still has watchers 2024-03-11T23:18:14.530 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-03-11T23:18:14.533 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:18:14.533 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:18:15.535 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_3 2024-03-11T23:18:15.535 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:18:15.535 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:18:15.535 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T23:18:15.535 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-03-11T23:18:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-03-11T23:18:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 0% complete...failed. 2024-03-11T23:18:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error: image still has watchers 2024-03-11T23:18:15.835 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-03-11T23:18:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:18:15.838 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 2 2024-03-11T23:18:16.371 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~54s 2024-03-11T23:18:17.842 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_3 2024-03-11T23:18:17.842 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:18:17.842 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:18:17.842 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T23:18:17.842 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-03-11T23:18:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-03-11T23:18:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 0% complete...failed. 2024-03-11T23:18:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error: image still has watchers 2024-03-11T23:18:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-03-11T23:18:17.936 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:18:17.936 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 4 2024-03-11T23:18:21.775 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~59s 2024-03-11T23:18:21.938 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_3 2024-03-11T23:18:21.938 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:18:21.938 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:18:21.938 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T23:18:21.938 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-03-11T23:18:21.987 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-03-11T23:18:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 0% complete...failed. 2024-03-11T23:18:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error: image still has watchers 2024-03-11T23:18:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-03-11T23:18:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:18:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 8 2024-03-11T23:18:27.262 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~65s 2024-03-11T23:18:30.037 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_3 2024-03-11T23:18:30.037 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:18:30.037 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:18:30.037 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T23:18:30.037 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-03-11T23:18:30.085 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-03-11T23:18:30.131 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 0% complete...failed. 2024-03-11T23:18:30.131 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error: image still has watchers 2024-03-11T23:18:30.131 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-03-11T23:18:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:18:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 16 2024-03-11T23:18:30.957 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7864190806768862) > weight (0.25) 2024-03-11T23:18:30.958 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6392985380757643) > weight (0.25) 2024-03-11T23:18:30.958 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9854447353589075) > weight (0.25) 2024-03-11T23:18:30.958 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-03-11T23:18:30.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-03-11T23:18:30.961 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-03-11T23:18:30.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi132.stderr:2024-03-11T23:18:30.959+0000 7ffb65156640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 59509) UID: 1000 2024-03-11T23:18:32.667 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~70s 2024-03-11T23:18:32.940 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5110487851685634) > weight (0.25) 2024-03-11T23:18:32.940 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.32963361435846383) > weight (0.25) 2024-03-11T23:18:32.940 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.5165022295870233) > weight (0.25) 2024-03-11T23:18:32.940 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.30819095113471884) > weight (0.25) 2024-03-11T23:18:32.940 INFO:tasks.rbd_mirror_thrash:waiting for 101 secs before thrashing 2024-03-11T23:18:38.071 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~76s 2024-03-11T23:18:39.962 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-03-11T23:18:39.963 INFO:teuthology.orchestra.run:waiting for 600 2024-03-11T23:18:39.963 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-03-11T23:18:39.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-03-11T23:18:39.963 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-03-11T23:18:39.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-03-11T23:18:39.966 INFO:tasks.rbd_mirror_thrash:waiting for 77 secs before thrashing 2024-03-11T23:18:43.475 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~81s 2024-03-11T23:18:46.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_3 2024-03-11T23:18:46.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:18:46.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:18:46.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_3 2024-03-11T23:18:46.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-03-11T23:18:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-03-11T23:18:46.366 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-03-11T23:18:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:18:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T23:18:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_4 2024-03-11T23:18:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_4 512M 2024-03-11T23:18:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:18:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:18:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:18:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:18:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T23:18:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:18:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:18:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 512M ']' 2024-03-11T23:18:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=512M 2024-03-11T23:18:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:18:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_4 2024-03-11T23:18:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_4 2024-03-11T23:18:46.416 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:18:46.416 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:18:46.416 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T23:18:46.416 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:18:46.416 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2024-03-11T23:18:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:18:47.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-03-11T23:18:47.377 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_4': 2024-03-11T23:18:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 512 MiB in 128 objects 2024-03-11T23:18:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:18:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:18:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 19a0d0d41bf2 2024-03-11T23:18:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.19a0d0d41bf2 2024-03-11T23:18:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:18:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:18:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:18:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:18:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:18:46 2024 2024-03-11T23:18:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:18:46 2024 2024-03-11T23:18:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:18:46 2024 2024-03-11T23:18:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:18:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:18:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 534b27e9-05a3-4e11-8e22-6fdc1c779e00 2024-03-11T23:18:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:18:47.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_4 2024-03-11T23:18:47.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:18:47.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:18:47.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T23:18:47.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:18:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=9 2024-03-11T23:18:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:18:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:18:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 2363392 bytes 2024-03-11T23:18:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 30755840 bytes 2024-03-11T23:18:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 139631616 bytes 2024-03-11T23:18:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 161003520 bytes 2024-03-11T23:18:48.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 177221632 bytes 2024-03-11T23:18:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 251573248 bytes 2024-03-11T23:18:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 329242624 bytes 2024-03-11T23:18:48.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 373512192 bytes 2024-03-11T23:18:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 436487168 bytes 2024-03-11T23:18:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 498764800 bytes 2024-03-11T23:18:48.879 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~87s 2024-03-11T23:18:48.925 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 556651520 bytes 2024-03-11T23:18:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 575205376 bytes 2024-03-11T23:18:49.185 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 655940608 bytes 2024-03-11T23:18:49.404 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 742421504 bytes 2024-03-11T23:18:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 810296320 bytes 2024-03-11T23:18:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 924580864 bytes 2024-03-11T23:18:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 941764608 bytes 2024-03-11T23:18:50.036 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 991145984 bytes 2024-03-11T23:18:50.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 1017315328 bytes 2024-03-11T23:18:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 1043681280 bytes 2024-03-11T23:18:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 1092410368 bytes 2024-03-11T23:18:50.483 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 1110535168 bytes 2024-03-11T23:18:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 1177981952 bytes 2024-03-11T23:18:50.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 1193166848 bytes 2024-03-11T23:18:51.090 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1248533504 bytes 2024-03-11T23:18:51.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1329200128 bytes 2024-03-11T23:18:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1371959296 bytes 2024-03-11T23:18:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1393196032 bytes 2024-03-11T23:18:51.674 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1451765760 bytes 2024-03-11T23:18:51.860 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1541763072 bytes 2024-03-11T23:18:51.984 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1582599168 bytes 2024-03-11T23:18:52.110 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1625714688 bytes 2024-03-11T23:18:52.227 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1657477120 bytes 2024-03-11T23:18:52.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1679526912 bytes 2024-03-11T23:18:52.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1729335296 bytes 2024-03-11T23:18:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1730816000 bytes 2024-03-11T23:18:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1752945664 bytes 2024-03-11T23:18:52.754 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1778508800 bytes 2024-03-11T23:18:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1804924928 bytes 2024-03-11T23:18:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1843032064 bytes 2024-03-11T23:18:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1912977408 bytes 2024-03-11T23:18:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1923544064 bytes 2024-03-11T23:18:53.382 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1982752768 bytes 2024-03-11T23:18:53.560 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 2062143488 bytes 2024-03-11T23:18:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 2138614784 bytes 2024-03-11T23:18:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 2204349440 bytes 2024-03-11T23:18:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 2233259008 bytes 2024-03-11T23:18:54.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 2270222336 bytes 2024-03-11T23:18:54.281 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~92s 2024-03-11T23:18:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 2293840896 bytes 2024-03-11T23:18:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 2333038592 bytes 2024-03-11T23:18:54.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5100 IOs, 2376241152 bytes 2024-03-11T23:18:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5200 IOs, 2421342208 bytes 2024-03-11T23:18:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5300 IOs, 2528229376 bytes 2024-03-11T23:18:55.014 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5400 IOs, 2528846848 bytes 2024-03-11T23:18:55.155 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5500 IOs, 2570728448 bytes 2024-03-11T23:18:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5600 IOs, 2628952064 bytes 2024-03-11T23:18:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5700 IOs, 2682963968 bytes 2024-03-11T23:18:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5800 IOs, 2708461568 bytes 2024-03-11T23:18:55.649 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5900 IOs, 2713631744 bytes 2024-03-11T23:18:55.770 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6000 IOs, 2745469952 bytes 2024-03-11T23:18:55.891 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6100 IOs, 2785987584 bytes 2024-03-11T23:18:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6200 IOs, 2814289920 bytes 2024-03-11T23:18:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6300 IOs, 2853235712 bytes 2024-03-11T23:18:56.274 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6400 IOs, 2914075648 bytes 2024-03-11T23:18:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:18:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:18:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:18:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:18:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2024-03-11T23:18:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:18:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:18:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T23:18:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=present 2024-03-11T23:18:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:18:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:18:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:18:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_4 2024-03-11T23:18:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:18:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:18:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_4 2024-03-11T23:18:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-03-11T23:18:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:18:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id=12b081f42f8 2024-03-11T23:18:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:18:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:18:56.537 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_4 present 12b081f42f8 2024-03-11T23:18:56.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:18:56.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:18:56.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T23:18:56.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=present 2024-03-11T23:18:56.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id=12b081f42f8 2024-03-11T23:18:56.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:18:56.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:18:56.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_4 2024-03-11T23:18:56.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:18:56.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:18:56.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_4 2024-03-11T23:18:56.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-03-11T23:18:56.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:18:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id=12b081f42f8 2024-03-11T23:18:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n 12b081f42f8 2024-03-11T23:18:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -z 12b081f42f8 -o 12b081f42f8 = 12b081f42f8 2024-03-11T23:18:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=present 2024-03-11T23:18:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test present = present 2024-03-11T23:18:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:18:56.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ disable_mirror cluster2 mirror image_4 2024-03-11T23:18:56.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:18:56.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:18:56.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T23:18:56.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2024-03-11T23:18:57.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi132.stderr:2024-03-11T23:18:57.434+0000 7f84b4efc640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55ba0f53f800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=188 2024-03-11T23:18:57.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi132.stderr:2024-03-11T23:18:57.434+0000 7f84b4efc640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55ba0f53f800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=188, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=baf1db9c-11db-489a-a01f-16a50bade1d5, primary_snap_id=bc, last_copied_object_number=128, snap_seqs={188=18446744073709551614}] 2024-03-11T23:18:59.685 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~97s 2024-03-11T23:19:02.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring disabled 2024-03-11T23:19:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-03-11T23:19:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:19:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:19:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T23:19:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:19:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:19:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:19:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:19:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_4 2024-03-11T23:19:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:19:02.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:19:02.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_4 2024-03-11T23:19:02.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-03-11T23:19:02.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:19:02.487 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-03-11T23:19:02.491 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:19:02.491 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:19:02.491 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:19:02.594 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-03-11T23:19:02.594 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:19:02.594 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:19:02.594 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T23:19:02.594 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:19:02.594 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:19:02.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:19:02.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:19:02.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_4 2024-03-11T23:19:02.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:19:02.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:19:02.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_4 2024-03-11T23:19:02.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-03-11T23:19:02.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:19:02.627 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-03-11T23:19:02.631 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:19:02.631 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:19:02.631 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:19:02.631 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:19:02.631 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ purge_snapshots cluster2 mirror image_4 2024-03-11T23:19:02.631 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:19:02.631 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:19:02.631 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T23:19:02.632 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2024-03-11T23:19:02.674 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_4 2024-03-11T23:19:02.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:19:02.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:19:02.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T23:19:02.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:19:02.675 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:19:02.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_4 2024-03-11T23:19:02.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:19:02.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:19:02.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T23:19:02.676 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-03-11T23:19:02.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-03-11T23:19:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 0% complete...failed. 2024-03-11T23:19:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error: image still has watchers 2024-03-11T23:19:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-03-11T23:19:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:19:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:19:03.764 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_4 2024-03-11T23:19:03.764 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:19:03.764 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:19:03.764 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T23:19:03.765 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-03-11T23:19:03.811 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-03-11T23:19:03.854 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 0% complete...failed. 2024-03-11T23:19:03.855 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error: image still has watchers 2024-03-11T23:19:03.855 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-03-11T23:19:03.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:19:03.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 2 2024-03-11T23:19:05.089 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~103s 2024-03-11T23:19:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_4 2024-03-11T23:19:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:19:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:19:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T23:19:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-03-11T23:19:05.909 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-03-11T23:19:05.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 0% complete...failed. 2024-03-11T23:19:05.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error: image still has watchers 2024-03-11T23:19:05.954 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-03-11T23:19:05.957 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:19:05.957 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 4 2024-03-11T23:19:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_4 2024-03-11T23:19:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:19:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:19:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T23:19:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-03-11T23:19:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-03-11T23:19:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 0% complete...failed. 2024-03-11T23:19:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error: image still has watchers 2024-03-11T23:19:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-03-11T23:19:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:19:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 8 2024-03-11T23:19:10.493 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~108s 2024-03-11T23:19:15.898 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~114s 2024-03-11T23:19:18.059 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_4 2024-03-11T23:19:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:19:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:19:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T23:19:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-03-11T23:19:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-03-11T23:19:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 0% complete...failed. 2024-03-11T23:19:18.154 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error: image still has watchers 2024-03-11T23:19:18.154 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-03-11T23:19:18.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:19:18.156 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 16 2024-03-11T23:19:21.302 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~119s 2024-03-11T23:19:26.707 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~124s 2024-03-11T23:19:32.110 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~130s 2024-03-11T23:19:34.158 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_4 2024-03-11T23:19:34.158 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:19:34.158 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:19:34.158 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_4 2024-03-11T23:19:34.158 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-03-11T23:19:34.208 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-03-11T23:19:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-03-11T23:19:34.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:19:34.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T23:19:34.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_5 2024-03-11T23:19:34.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_5 512M 2024-03-11T23:19:34.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:19:34.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:19:34.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:19:34.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:19:34.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T23:19:34.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:19:34.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:19:34.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 512M ']' 2024-03-11T23:19:34.398 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=512M 2024-03-11T23:19:34.398 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:19:34.398 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_5 2024-03-11T23:19:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_5 2024-03-11T23:19:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:19:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:19:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T23:19:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:19:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2024-03-11T23:19:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:19:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-03-11T23:19:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_5': 2024-03-11T23:19:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 512 MiB in 128 objects 2024-03-11T23:19:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:19:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:19:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 19d62402c031 2024-03-11T23:19:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.19d62402c031 2024-03-11T23:19:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:19:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:19:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:19:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:19:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:19:34 2024 2024-03-11T23:19:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:19:34 2024 2024-03-11T23:19:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:19:34 2024 2024-03-11T23:19:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:19:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:19:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: a6abe196-d710-40d8-9e64-8a0bd1917d42 2024-03-11T23:19:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:19:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_5 2024-03-11T23:19:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:19:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:19:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T23:19:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:19:35.165 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=11 2024-03-11T23:19:35.165 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:19:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:19:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 25380864 bytes 2024-03-11T23:19:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 67206144 bytes 2024-03-11T23:19:35.550 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 97107968 bytes 2024-03-11T23:19:35.690 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 152221696 bytes 2024-03-11T23:19:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 234828800 bytes 2024-03-11T23:19:35.960 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 250766336 bytes 2024-03-11T23:19:36.114 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 318966784 bytes 2024-03-11T23:19:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 369118208 bytes 2024-03-11T23:19:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 383740928 bytes 2024-03-11T23:19:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 414360576 bytes 2024-03-11T23:19:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 464605184 bytes 2024-03-11T23:19:36.746 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 514820096 bytes 2024-03-11T23:19:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 588150784 bytes 2024-03-11T23:19:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 592744448 bytes 2024-03-11T23:19:37.103 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 647644160 bytes 2024-03-11T23:19:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 681201664 bytes 2024-03-11T23:19:37.327 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 705799168 bytes 2024-03-11T23:19:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 773038080 bytes 2024-03-11T23:19:37.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~135s 2024-03-11T23:19:37.586 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 810055680 bytes 2024-03-11T23:19:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 857655296 bytes 2024-03-11T23:19:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 942357504 bytes 2024-03-11T23:19:37.992 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 971915264 bytes 2024-03-11T23:19:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 1016450048 bytes 2024-03-11T23:19:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 1078902784 bytes 2024-03-11T23:19:38.424 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1132596224 bytes 2024-03-11T23:19:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1174451200 bytes 2024-03-11T23:19:38.672 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1196497920 bytes 2024-03-11T23:19:38.783 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1229882368 bytes 2024-03-11T23:19:38.875 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1251189760 bytes 2024-03-11T23:19:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1270268928 bytes 2024-03-11T23:19:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1285890048 bytes 2024-03-11T23:19:39.208 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1361510400 bytes 2024-03-11T23:19:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1385190400 bytes 2024-03-11T23:19:39.438 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1399252992 bytes 2024-03-11T23:19:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1433948160 bytes 2024-03-11T23:19:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1469763584 bytes 2024-03-11T23:19:39.822 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1544878080 bytes 2024-03-11T23:19:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1578408960 bytes 2024-03-11T23:19:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1601131520 bytes 2024-03-11T23:19:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1669302272 bytes 2024-03-11T23:19:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1689441280 bytes 2024-03-11T23:19:40.364 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr:2024-03-11T23:19:40.360+0000 7f4dc2287640 -1 reset not still connected to 0x55f1a62a1e10 2024-03-11T23:19:40.364 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr:2024-03-11T23:19:40.360+0000 7f4dc2287640 -1 reset not still connected to 0x55f1c05f9ba0 2024-03-11T23:19:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1725898752 bytes 2024-03-11T23:19:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1777967104 bytes 2024-03-11T23:19:40.739 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 1832220672 bytes 2024-03-11T23:19:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 1924470784 bytes 2024-03-11T23:19:41.029 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 1939924992 bytes 2024-03-11T23:19:41.107 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 1959603200 bytes 2024-03-11T23:19:41.276 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 2008983552 bytes 2024-03-11T23:19:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 2057669632 bytes 2024-03-11T23:19:41.524 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 2070525952 bytes 2024-03-11T23:19:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5100 IOs, 2111808512 bytes 2024-03-11T23:19:41.761 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5200 IOs, 2136579072 bytes 2024-03-11T23:19:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5300 IOs, 2207154176 bytes 2024-03-11T23:19:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5400 IOs, 2207750144 bytes 2024-03-11T23:19:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5500 IOs, 2276859904 bytes 2024-03-11T23:19:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5600 IOs, 2302456832 bytes 2024-03-11T23:19:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5700 IOs, 2308071424 bytes 2024-03-11T23:19:42.510 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5800 IOs, 2390305792 bytes 2024-03-11T23:19:42.664 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5900 IOs, 2444299264 bytes 2024-03-11T23:19:42.789 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6000 IOs, 2495872000 bytes 2024-03-11T23:19:42.899 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6100 IOs, 2514625536 bytes 2024-03-11T23:19:42.918 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~141s 2024-03-11T23:19:42.987 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6200 IOs, 2524930048 bytes 2024-03-11T23:19:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6300 IOs, 2576979968 bytes 2024-03-11T23:19:43.286 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6400 IOs, 2654294016 bytes 2024-03-11T23:19:43.426 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6500 IOs, 2695668736 bytes 2024-03-11T23:19:43.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6600 IOs, 2774622208 bytes 2024-03-11T23:19:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6700 IOs, 2775595008 bytes 2024-03-11T23:19:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6800 IOs, 2824135680 bytes 2024-03-11T23:19:44.145 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6900 IOs, 2886540288 bytes 2024-03-11T23:19:44.285 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7000 IOs, 2908136448 bytes 2024-03-11T23:19:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7100 IOs, 2961360896 bytes 2024-03-11T23:19:44.579 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7200 IOs, 3031490560 bytes 2024-03-11T23:19:44.695 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7300 IOs, 3057926144 bytes 2024-03-11T23:19:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7400 IOs, 3091748864 bytes 2024-03-11T23:19:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7500 IOs, 3122728960 bytes 2024-03-11T23:19:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7600 IOs, 3223442432 bytes 2024-03-11T23:19:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7700 IOs, 3247004672 bytes 2024-03-11T23:19:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7800 IOs, 3286489088 bytes 2024-03-11T23:19:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:7900 IOs, 3322565632 bytes 2024-03-11T23:19:45.626 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:8000 IOs, 3379944448 bytes 2024-03-11T23:19:45.794 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:8100 IOs, 3456227328 bytes 2024-03-11T23:19:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:8200 IOs, 3531929600 bytes 2024-03-11T23:19:46.069 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:8300 IOs, 3552257024 bytes 2024-03-11T23:19:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:19:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:19:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:19:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:19:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2024-03-11T23:19:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:19:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:19:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T23:19:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=present 2024-03-11T23:19:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:19:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:19:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:19:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_5 2024-03-11T23:19:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:19:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:19:46.172 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_5 2024-03-11T23:19:46.172 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-03-11T23:19:46.172 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:19:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id=12b04e4ad75f 2024-03-11T23:19:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:19:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:19:46.314 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_5 present 12b04e4ad75f 2024-03-11T23:19:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:19:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:19:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T23:19:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=present 2024-03-11T23:19:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id=12b04e4ad75f 2024-03-11T23:19:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:19:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:19:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_5 2024-03-11T23:19:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:19:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:19:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_5 2024-03-11T23:19:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-03-11T23:19:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:19:46.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id=12b04e4ad75f 2024-03-11T23:19:46.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n 12b04e4ad75f 2024-03-11T23:19:46.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -z 12b04e4ad75f -o 12b04e4ad75f = 12b04e4ad75f 2024-03-11T23:19:46.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=present 2024-03-11T23:19:46.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test present = present 2024-03-11T23:19:46.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:19:46.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ disable_mirror cluster2 mirror image_5 2024-03-11T23:19:46.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:19:46.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:19:46.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T23:19:46.356 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2024-03-11T23:19:47.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi132.stderr:2024-03-11T23:19:47.176+0000 7f84b4efc640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55ba0ef9d800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=189 2024-03-11T23:19:47.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi132.stderr:2024-03-11T23:19:47.176+0000 7f84b4efc640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55ba0ef9d800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=189, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=baf1db9c-11db-489a-a01f-16a50bade1d5, primary_snap_id=bd, last_copied_object_number=128, snap_seqs={189=18446744073709551614}] 2024-03-11T23:19:48.323 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~146s 2024-03-11T23:19:52.182 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring disabled 2024-03-11T23:19:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-03-11T23:19:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:19:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:19:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T23:19:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:19:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:19:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:19:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:19:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_5 2024-03-11T23:19:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:19:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:19:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_5 2024-03-11T23:19:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-03-11T23:19:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:19:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-03-11T23:19:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:19:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:19:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:19:52.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-03-11T23:19:52.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:19:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:19:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T23:19:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:19:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:19:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:19:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:19:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_5 2024-03-11T23:19:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:19:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:19:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_5 2024-03-11T23:19:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-03-11T23:19:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:19:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-03-11T23:19:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:19:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:19:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:19:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:19:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ purge_snapshots cluster2 mirror image_5 2024-03-11T23:19:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:19:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:19:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T23:19:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2024-03-11T23:19:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_5 2024-03-11T23:19:52.422 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:19:52.422 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:19:52.422 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T23:19:52.422 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:19:52.422 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:19:52.422 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_5 2024-03-11T23:19:52.422 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:19:52.422 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:19:52.422 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T23:19:52.422 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-03-11T23:19:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-03-11T23:19:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 0% complete...failed. 2024-03-11T23:19:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error: image still has watchers 2024-03-11T23:19:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-03-11T23:19:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:19:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:19:53.509 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_5 2024-03-11T23:19:53.509 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:19:53.509 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:19:53.509 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T23:19:53.510 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-03-11T23:19:53.558 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-03-11T23:19:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 0% complete...failed. 2024-03-11T23:19:53.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error: image still has watchers 2024-03-11T23:19:53.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-03-11T23:19:53.605 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:19:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 2 2024-03-11T23:19:53.727 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~151s 2024-03-11T23:19:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_5 2024-03-11T23:19:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:19:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:19:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T23:19:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-03-11T23:19:55.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-03-11T23:19:55.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 0% complete...failed. 2024-03-11T23:19:55.702 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error: image still has watchers 2024-03-11T23:19:55.702 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-03-11T23:19:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:19:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 4 2024-03-11T23:19:56.967 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8544725605708405) > weight (0.25) 2024-03-11T23:19:56.967 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.852410196377717) > weight (0.25) 2024-03-11T23:19:56.967 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.44318680986463654) > weight (0.25) 2024-03-11T23:19:56.967 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.6914855568614628) > weight (0.25) 2024-03-11T23:19:56.968 INFO:tasks.rbd_mirror_thrash:waiting for 93 secs before thrashing 2024-03-11T23:19:59.131 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~157s 2024-03-11T23:19:59.706 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_5 2024-03-11T23:19:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:19:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:19:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T23:19:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-03-11T23:19:59.754 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-03-11T23:19:59.800 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 0% complete...failed. 2024-03-11T23:19:59.800 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error: image still has watchers 2024-03-11T23:19:59.800 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-03-11T23:19:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:19:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 8 2024-03-11T23:20:04.535 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~162s 2024-03-11T23:20:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_5 2024-03-11T23:20:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:20:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:20:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T23:20:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-03-11T23:20:07.853 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-03-11T23:20:07.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 0% complete...failed. 2024-03-11T23:20:07.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error: image still has watchers 2024-03-11T23:20:07.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-03-11T23:20:07.900 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:20:07.900 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 16 2024-03-11T23:20:09.939 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~168s 2024-03-11T23:20:13.941 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.26108307299045164) > weight (0.25) 2024-03-11T23:20:13.941 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8011830641392669) > weight (0.25) 2024-03-11T23:20:13.941 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.4346663840330274) > weight (0.25) 2024-03-11T23:20:13.941 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-03-11T23:20:13.942 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-03-11T23:20:13.942 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-03-11T23:20:13.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi132.stderr:2024-03-11T23:20:13.940+0000 7ff3a2556640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 74366) UID: 1000 2024-03-11T23:20:15.344 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~173s 2024-03-11T23:20:15.943 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-03-11T23:20:15.943 INFO:teuthology.orchestra.run:waiting for 600 2024-03-11T23:20:15.944 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-03-11T23:20:15.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-03-11T23:20:15.944 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-03-11T23:20:15.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-03-11T23:20:15.947 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2024-03-11T23:20:20.749 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~178s 2024-03-11T23:20:23.901 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_5 2024-03-11T23:20:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:20:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:20:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_5 2024-03-11T23:20:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-03-11T23:20:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-03-11T23:20:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-03-11T23:20:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:20:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T23:20:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_6 2024-03-11T23:20:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_6 512M 2024-03-11T23:20:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:20:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:20:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:20:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:20:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:20:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:20:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:20:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 512M ']' 2024-03-11T23:20:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=512M 2024-03-11T23:20:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:20:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_6 2024-03-11T23:20:24.176 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_6 2024-03-11T23:20:24.176 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:20:24.176 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:20:24.176 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:20:24.176 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:20:24.176 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2024-03-11T23:20:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:20:24.403 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-03-11T23:20:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_6': 2024-03-11T23:20:24.440 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 512 MiB in 128 objects 2024-03-11T23:20:24.440 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:20:24.440 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:20:24.440 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 1a167894a9b1 2024-03-11T23:20:24.440 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.1a167894a9b1 2024-03-11T23:20:24.440 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:20:24.440 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:20:24.440 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:20:24.440 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:20:24.440 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:20:24 2024 2024-03-11T23:20:24.440 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:20:24 2024 2024-03-11T23:20:24.440 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:20:24 2024 2024-03-11T23:20:24.440 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:20:24.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:20:24.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 8839aafc-67fe-4ed5-986e-310ddaf82a66 2024-03-11T23:20:24.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:20:24.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_6 2024-03-11T23:20:24.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:20:24.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:20:24.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:20:24.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:20:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=9 2024-03-11T23:20:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:20:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:20:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 26788864 bytes 2024-03-11T23:20:24.695 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 82754560 bytes 2024-03-11T23:20:24.818 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 125460480 bytes 2024-03-11T23:20:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 151018496 bytes 2024-03-11T23:20:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 159835136 bytes 2024-03-11T23:20:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 190950400 bytes 2024-03-11T23:20:25.219 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 218202112 bytes 2024-03-11T23:20:25.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 274620416 bytes 2024-03-11T23:20:25.507 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 309986304 bytes 2024-03-11T23:20:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 333031424 bytes 2024-03-11T23:20:25.742 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 375974912 bytes 2024-03-11T23:20:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 404694016 bytes 2024-03-11T23:20:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 445648896 bytes 2024-03-11T23:20:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 460104704 bytes 2024-03-11T23:20:26.152 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~184s 2024-03-11T23:20:26.243 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 567160832 bytes 2024-03-11T23:20:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 593894400 bytes 2024-03-11T23:20:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 657415168 bytes 2024-03-11T23:20:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 701990912 bytes 2024-03-11T23:20:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 702607360 bytes 2024-03-11T23:20:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 770754560 bytes 2024-03-11T23:20:26.935 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 784204800 bytes 2024-03-11T23:20:26.955 INFO:tasks.ceph.cluster2.osd.2.smithi132.stderr:2024-03-11T23:20:26.952+0000 7f6376022640 -1 reset not still connected to 0x5650809aaf70 2024-03-11T23:20:26.956 INFO:tasks.ceph.cluster2.osd.2.smithi132.stderr:2024-03-11T23:20:26.952+0000 7f6376022640 -1 reset not still connected to 0x565090c2a4e0 2024-03-11T23:20:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 886292480 bytes 2024-03-11T23:20:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 897751040 bytes 2024-03-11T23:20:27.334 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 920713216 bytes 2024-03-11T23:20:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 975349760 bytes 2024-03-11T23:20:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1047166976 bytes 2024-03-11T23:20:27.757 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1081070592 bytes 2024-03-11T23:20:27.866 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1122022400 bytes 2024-03-11T23:20:27.956 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1128273920 bytes 2024-03-11T23:20:28.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1133645824 bytes 2024-03-11T23:20:28.363 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1183628288 bytes 2024-03-11T23:20:28.460 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1195022336 bytes 2024-03-11T23:20:28.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1218972672 bytes 2024-03-11T23:20:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1256645632 bytes 2024-03-11T23:20:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1306730496 bytes 2024-03-11T23:20:28.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1403087872 bytes 2024-03-11T23:20:29.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1456986112 bytes 2024-03-11T23:20:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1539213312 bytes 2024-03-11T23:20:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1637072896 bytes 2024-03-11T23:20:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1652196352 bytes 2024-03-11T23:20:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1673229312 bytes 2024-03-11T23:20:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1742521344 bytes 2024-03-11T23:20:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1795146752 bytes 2024-03-11T23:20:30.123 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 1798325248 bytes 2024-03-11T23:20:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 1818700800 bytes 2024-03-11T23:20:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 1863489536 bytes 2024-03-11T23:20:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 1973590016 bytes 2024-03-11T23:20:30.734 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 2053737472 bytes 2024-03-11T23:20:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 2059973632 bytes 2024-03-11T23:20:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 2133223424 bytes 2024-03-11T23:20:31.073 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5100 IOs, 2133799936 bytes 2024-03-11T23:20:31.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5200 IOs, 2209931264 bytes 2024-03-11T23:20:31.386 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5300 IOs, 2285978624 bytes 2024-03-11T23:20:31.549 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5400 IOs, 2353579008 bytes 2024-03-11T23:20:31.556 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~189s 2024-03-11T23:20:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5500 IOs, 2390675456 bytes 2024-03-11T23:20:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5600 IOs, 2470325248 bytes 2024-03-11T23:20:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5700 IOs, 2499295232 bytes 2024-03-11T23:20:32.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5800 IOs, 2588568576 bytes 2024-03-11T23:20:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5900 IOs, 2612988928 bytes 2024-03-11T23:20:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6000 IOs, 2701231104 bytes 2024-03-11T23:20:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6100 IOs, 2761787392 bytes 2024-03-11T23:20:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6200 IOs, 2880407552 bytes 2024-03-11T23:20:33.011 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6300 IOs, 2956409856 bytes 2024-03-11T23:20:33.147 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6400 IOs, 3016365056 bytes 2024-03-11T23:20:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6500 IOs, 3042340864 bytes 2024-03-11T23:20:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:6600 IOs, 3058191360 bytes 2024-03-11T23:20:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:20:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:20:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:20:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:20:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2024-03-11T23:20:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:20:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:20:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:20:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=present 2024-03-11T23:20:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:20:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:20:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:20:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_6 2024-03-11T23:20:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:20:33.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:20:33.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_6 2024-03-11T23:20:33.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-03-11T23:20:33.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:20:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id=12b01f75f6fe 2024-03-11T23:20:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:20:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:20:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_6 present 12b01f75f6fe 2024-03-11T23:20:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:20:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:20:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:20:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=present 2024-03-11T23:20:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id=12b01f75f6fe 2024-03-11T23:20:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:20:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:20:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_6 2024-03-11T23:20:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:20:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:20:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_6 2024-03-11T23:20:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-03-11T23:20:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:20:33.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id=12b01f75f6fe 2024-03-11T23:20:33.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n 12b01f75f6fe 2024-03-11T23:20:33.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -z 12b01f75f6fe -o 12b01f75f6fe = 12b01f75f6fe 2024-03-11T23:20:33.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=present 2024-03-11T23:20:33.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test present = present 2024-03-11T23:20:33.637 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:20:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ disable_mirror cluster2 mirror image_6 2024-03-11T23:20:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:20:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:20:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:20:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2024-03-11T23:20:34.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi132.stderr:2024-03-11T23:20:34.449+0000 7f84b4efc640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55ba0f736800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=190 2024-03-11T23:20:34.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi132.stderr:2024-03-11T23:20:34.449+0000 7f84b4efc640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55ba0f736800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=190, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=baf1db9c-11db-489a-a01f-16a50bade1d5, primary_snap_id=be, last_copied_object_number=128, snap_seqs={190=18446744073709551614}] 2024-03-11T23:20:36.960 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~195s 2024-03-11T23:20:39.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring disabled 2024-03-11T23:20:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-03-11T23:20:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:20:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:20:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:20:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:20:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:20:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:20:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:20:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_6 2024-03-11T23:20:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:20:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:20:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_6 2024-03-11T23:20:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-03-11T23:20:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:20:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-03-11T23:20:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:20:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:20:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:20:39.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-03-11T23:20:39.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:20:39.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:20:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:20:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:20:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:20:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:20:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:20:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_6 2024-03-11T23:20:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:20:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:20:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_6 2024-03-11T23:20:39.602 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-03-11T23:20:39.602 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:20:39.635 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-03-11T23:20:39.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:20:39.639 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:20:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:20:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:20:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ purge_snapshots cluster2 mirror image_6 2024-03-11T23:20:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:20:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:20:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:20:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2024-03-11T23:20:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_6 2024-03-11T23:20:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:20:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:20:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:20:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:20:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:20:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_6 2024-03-11T23:20:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:20:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:20:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:20:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-03-11T23:20:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-03-11T23:20:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 0% complete...failed. 2024-03-11T23:20:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error: image still has watchers 2024-03-11T23:20:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-03-11T23:20:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:20:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:20:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_6 2024-03-11T23:20:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:20:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:20:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:20:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-03-11T23:20:40.818 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-03-11T23:20:40.862 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 0% complete...failed. 2024-03-11T23:20:40.862 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error: image still has watchers 2024-03-11T23:20:40.862 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-03-11T23:20:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:20:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 2 2024-03-11T23:20:42.364 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~200s 2024-03-11T23:20:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_6 2024-03-11T23:20:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:20:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:20:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:20:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-03-11T23:20:42.914 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-03-11T23:20:42.960 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 0% complete...failed. 2024-03-11T23:20:42.960 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error: image still has watchers 2024-03-11T23:20:42.960 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-03-11T23:20:42.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:20:42.963 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 4 2024-03-11T23:20:46.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_6 2024-03-11T23:20:46.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:20:46.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:20:46.964 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:20:46.965 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-03-11T23:20:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-03-11T23:20:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 0% complete...failed. 2024-03-11T23:20:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error: image still has watchers 2024-03-11T23:20:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-03-11T23:20:47.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:20:47.060 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 8 2024-03-11T23:20:47.768 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~205s 2024-03-11T23:20:53.172 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~211s 2024-03-11T23:20:55.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_6 2024-03-11T23:20:55.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:20:55.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:20:55.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:20:55.062 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-03-11T23:20:55.114 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-03-11T23:20:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 0% complete...failed. 2024-03-11T23:20:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error: image still has watchers 2024-03-11T23:20:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-03-11T23:20:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:20:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 16 2024-03-11T23:20:58.576 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~216s 2024-03-11T23:21:03.980 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~222s 2024-03-11T23:21:09.384 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~227s 2024-03-11T23:21:11.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_6 2024-03-11T23:21:11.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:21:11.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:21:11.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_6 2024-03-11T23:21:11.164 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-03-11T23:21:11.213 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-03-11T23:21:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-03-11T23:21:11.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:21:11.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T23:21:11.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_7 2024-03-11T23:21:11.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_7 512M 2024-03-11T23:21:11.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:21:11.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:21:11.598 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:21:11.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:21:11.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:21:11.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:21:11.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:21:11.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 512M ']' 2024-03-11T23:21:11.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=512M 2024-03-11T23:21:11.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:21:11.599 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_7 2024-03-11T23:21:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_7 2024-03-11T23:21:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:21:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:21:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:21:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:21:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2024-03-11T23:21:12.128 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:21:12.133 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-03-11T23:21:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_7': 2024-03-11T23:21:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 512 MiB in 128 objects 2024-03-11T23:21:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:21:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:21:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 1a4c89b633bc 2024-03-11T23:21:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.1a4c89b633bc 2024-03-11T23:21:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:21:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:21:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:21:12.169 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:21:12.169 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:21:11 2024 2024-03-11T23:21:12.169 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:21:11 2024 2024-03-11T23:21:12.169 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:21:11 2024 2024-03-11T23:21:12.169 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:21:12.169 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:21:12.169 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 898aeff2-075a-4795-a6bd-30badb0fc1fb 2024-03-11T23:21:12.169 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:21:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_7 2024-03-11T23:21:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:21:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:21:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:21:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:21:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=7 2024-03-11T23:21:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:21:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:21:12.281 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 1018880 bytes 2024-03-11T23:21:12.465 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 117566464 bytes 2024-03-11T23:21:12.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 190966784 bytes 2024-03-11T23:21:12.718 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 191563776 bytes 2024-03-11T23:21:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 241489920 bytes 2024-03-11T23:21:12.967 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 276489216 bytes 2024-03-11T23:21:13.078 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 307761152 bytes 2024-03-11T23:21:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 401296384 bytes 2024-03-11T23:21:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 402244608 bytes 2024-03-11T23:21:13.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 454779904 bytes 2024-03-11T23:21:13.591 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 494793728 bytes 2024-03-11T23:21:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 544491520 bytes 2024-03-11T23:21:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 588713984 bytes 2024-03-11T23:21:14.073 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 664346624 bytes 2024-03-11T23:21:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 698225664 bytes 2024-03-11T23:21:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 701406208 bytes 2024-03-11T23:21:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 720432128 bytes 2024-03-11T23:21:14.448 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 727597056 bytes 2024-03-11T23:21:14.518 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 728237056 bytes 2024-03-11T23:21:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 799851520 bytes 2024-03-11T23:21:14.773 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 830301184 bytes 2024-03-11T23:21:14.787 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~232s 2024-03-11T23:21:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 858255360 bytes 2024-03-11T23:21:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 893462528 bytes 2024-03-11T23:21:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 958289920 bytes 2024-03-11T23:21:15.228 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 976303104 bytes 2024-03-11T23:21:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1054073856 bytes 2024-03-11T23:21:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1140704256 bytes 2024-03-11T23:21:15.678 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1155758080 bytes 2024-03-11T23:21:15.782 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1190564864 bytes 2024-03-11T23:21:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1228745728 bytes 2024-03-11T23:21:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1257667584 bytes 2024-03-11T23:21:16.147 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1328444416 bytes 2024-03-11T23:21:16.286 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1380368384 bytes 2024-03-11T23:21:16.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1413511168 bytes 2024-03-11T23:21:16.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1460088832 bytes 2024-03-11T23:21:16.615 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1520659456 bytes 2024-03-11T23:21:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1522766848 bytes 2024-03-11T23:21:16.837 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1572714496 bytes 2024-03-11T23:21:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1591530496 bytes 2024-03-11T23:21:17.176 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1723006976 bytes 2024-03-11T23:21:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1786043392 bytes 2024-03-11T23:21:17.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1823047680 bytes 2024-03-11T23:21:17.572 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1874296832 bytes 2024-03-11T23:21:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 1918180352 bytes 2024-03-11T23:21:17.779 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 1935312896 bytes 2024-03-11T23:21:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 1974141952 bytes 2024-03-11T23:21:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 2006695936 bytes 2024-03-11T23:21:18.135 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 2066709504 bytes 2024-03-11T23:21:18.273 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4900 IOs, 2114222080 bytes 2024-03-11T23:21:18.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5000 IOs, 2217568256 bytes 2024-03-11T23:21:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5100 IOs, 2335959040 bytes 2024-03-11T23:21:18.842 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5200 IOs, 2423855104 bytes 2024-03-11T23:21:18.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5300 IOs, 2468255744 bytes 2024-03-11T23:21:19.022 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5400 IOs, 2468907008 bytes 2024-03-11T23:21:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:5500 IOs, 2496076800 bytes 2024-03-11T23:21:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:21:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:21:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:21:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:21:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2024-03-11T23:21:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:21:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:21:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:21:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=present 2024-03-11T23:21:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:21:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:21:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:21:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_7 2024-03-11T23:21:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:21:19.180 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:21:19.180 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_7 2024-03-11T23:21:19.180 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-03-11T23:21:19.180 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:21:19.223 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id=12b0111a2f33 2024-03-11T23:21:19.223 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:21:19.223 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:21:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_7 present 12b0111a2f33 2024-03-11T23:21:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:21:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:21:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:21:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=present 2024-03-11T23:21:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id=12b0111a2f33 2024-03-11T23:21:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:21:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:21:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_7 2024-03-11T23:21:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:21:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:21:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_7 2024-03-11T23:21:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-03-11T23:21:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:21:19.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id=12b0111a2f33 2024-03-11T23:21:19.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n 12b0111a2f33 2024-03-11T23:21:19.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -z 12b0111a2f33 -o 12b0111a2f33 = 12b0111a2f33 2024-03-11T23:21:19.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=present 2024-03-11T23:21:19.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test present = present 2024-03-11T23:21:19.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:21:19.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ disable_mirror cluster2 mirror image_7 2024-03-11T23:21:19.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:21:19.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:21:19.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:21:19.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_7 2024-03-11T23:21:20.192 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~238s 2024-03-11T23:21:20.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi132.stderr:2024-03-11T23:21:20.191+0000 7f84b4efc640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55ba0f427800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=191 2024-03-11T23:21:20.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi132.stderr:2024-03-11T23:21:20.191+0000 7f84b4efc640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55ba0f427800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=191, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=baf1db9c-11db-489a-a01f-16a50bade1d5, primary_snap_id=bf, last_copied_object_number=128, snap_seqs={191=18446744073709551614}] 2024-03-11T23:21:23.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi132.stderr:2024-03-11T23:21:23.087+0000 7f4ea480a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556662a224e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:21:23.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi132.stderr:2024-03-11T23:21:23.087+0000 7f4eab017640 -1 rbd::mirror::ImageReplayer: 0x556662283180 [3/898aeff2-075a-4795-a6bd-30badb0fc1fb] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:21:24.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi132.stderr:2024-03-11T23:21:24.117+0000 7f4ea4009640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556662a22680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:21:24.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi132.stderr:2024-03-11T23:21:24.117+0000 7f4eab017640 -1 rbd::mirror::ImageReplayer: 0x556662283180 [3/898aeff2-075a-4795-a6bd-30badb0fc1fb] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:21:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring disabled 2024-03-11T23:21:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-03-11T23:21:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:21:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:21:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:21:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=deleted 2024-03-11T23:21:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:21:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:21:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:21:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_7 2024-03-11T23:21:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:21:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:21:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_7 2024-03-11T23:21:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-03-11T23:21:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:21:25.236 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-03-11T23:21:25.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id= 2024-03-11T23:21:25.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:21:25.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:21:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-03-11T23:21:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:21:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:21:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:21:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=deleted 2024-03-11T23:21:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:21:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:21:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:21:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_7 2024-03-11T23:21:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:21:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:21:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_7 2024-03-11T23:21:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-03-11T23:21:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:21:25.377 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-03-11T23:21:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id= 2024-03-11T23:21:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:21:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test deleted = deleted 2024-03-11T23:21:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:21:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ purge_snapshots cluster2 mirror image_7 2024-03-11T23:21:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:21:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:21:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:21:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 snap purge mirror/image_7 2024-03-11T23:21:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image_retry cluster2 mirror image_7 2024-03-11T23:21:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:21:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:21:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:21:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:21:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0 2024-03-11T23:21:25.425 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_7 2024-03-11T23:21:25.425 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:21:25.425 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:21:25.425 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:21:25.425 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-03-11T23:21:25.468 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-03-11T23:21:25.509 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 0% complete...failed. 2024-03-11T23:21:25.509 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error: image still has watchers 2024-03-11T23:21:25.509 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-03-11T23:21:25.512 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:21:25.512 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 1 2024-03-11T23:21:25.597 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~243s 2024-03-11T23:21:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_7 2024-03-11T23:21:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:21:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:21:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:21:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-03-11T23:21:26.557 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-03-11T23:21:26.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 0% complete...failed. 2024-03-11T23:21:26.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error: image still has watchers 2024-03-11T23:21:26.600 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-03-11T23:21:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:21:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 2 2024-03-11T23:21:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_7 2024-03-11T23:21:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:21:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:21:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:21:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-03-11T23:21:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-03-11T23:21:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 0% complete...failed. 2024-03-11T23:21:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error: image still has watchers 2024-03-11T23:21:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-03-11T23:21:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:21:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 4 2024-03-11T23:21:29.968 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-03-11T23:21:29.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-03-11T23:21:29.969 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-03-11T23:21:29.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi132.stderr:2024-03-11T23:21:29.967+0000 7f251c138640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 65642) UID: 1000 2024-03-11T23:21:31.001 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~249s 2024-03-11T23:21:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_7 2024-03-11T23:21:32.702 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:21:32.702 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:21:32.702 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:21:32.702 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-03-11T23:21:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-03-11T23:21:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 0% complete...failed. 2024-03-11T23:21:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error: image still has watchers 2024-03-11T23:21:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-03-11T23:21:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:21:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 8 2024-03-11T23:21:36.405 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~254s 2024-03-11T23:21:37.970 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-03-11T23:21:37.970 INFO:teuthology.orchestra.run:waiting for 600 2024-03-11T23:21:37.971 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-03-11T23:21:37.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-03-11T23:21:37.972 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-03-11T23:21:37.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-03-11T23:21:37.975 INFO:tasks.rbd_mirror_thrash:waiting for 93 secs before thrashing 2024-03-11T23:21:38.950 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4950692669758354) > weight (0.25) 2024-03-11T23:21:38.950 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5058246468437186) > weight (0.25) 2024-03-11T23:21:38.950 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-03-11T23:21:38.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-03-11T23:21:38.950 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-03-11T23:21:38.951 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-03-11T23:21:38.951 INFO:teuthology.orchestra.run:waiting for 600 2024-03-11T23:21:38.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi132.stderr:2024-03-11T23:21:38.949+0000 7f1e3489b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 60055) UID: 1000 2024-03-11T23:21:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_7 2024-03-11T23:21:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:21:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:21:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:21:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-03-11T23:21:40.850 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-03-11T23:21:40.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 0% complete...failed. 2024-03-11T23:21:40.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: error: image still has watchers 2024-03-11T23:21:40.894 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-03-11T23:21:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-11T23:21:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 16 2024-03-11T23:21:41.809 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~259s 2024-03-11T23:21:45.053 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-03-11T23:21:45.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-03-11T23:21:45.053 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-03-11T23:21:45.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-03-11T23:21:45.056 INFO:tasks.rbd_mirror_thrash:waiting for 99 secs before thrashing 2024-03-11T23:21:47.212 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~265s 2024-03-11T23:21:52.615 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~270s 2024-03-11T23:21:56.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ remove_image cluster2 mirror image_7 2024-03-11T23:21:56.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:21:56.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:21:56.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_7 2024-03-11T23:21:56.898 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-03-11T23:21:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-03-11T23:21:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-03-11T23:21:57.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:21:57.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for i in `seq 1 10` 2024-03-11T23:21:57.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image=image_8 2024-03-11T23:21:57.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ create_image cluster2 mirror image_8 512M 2024-03-11T23:21:57.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:21:57.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:21:57.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:21:57.141 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:21:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_8 2024-03-11T23:21:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:21:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local size=128 2024-03-11T23:21:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' -n 512M ']' 2024-03-11T23:21:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ size=512M 2024-03-11T23:21:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ shift 2024-03-11T23:21:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_8 2024-03-11T23:21:57.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ enable_mirror cluster2 mirror image_8 2024-03-11T23:21:57.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:21:57.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:21:57.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_8 2024-03-11T23:21:57.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local mode=snapshot 2024-03-11T23:21:57.186 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2024-03-11T23:21:58.020 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~276s 2024-03-11T23:21:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:Mirroring enabled 2024-03-11T23:21:58.150 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-03-11T23:21:58.189 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:rbd image 'image_8': 2024-03-11T23:21:58.189 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: size 512 MiB in 128 objects 2024-03-11T23:21:58.189 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: order 22 (4 MiB objects) 2024-03-11T23:21:58.189 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: snapshot_count: 1 2024-03-11T23:21:58.189 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: id: 1a8fd3c84d77 2024-03-11T23:21:58.189 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: block_name_prefix: rbd_data.1a8fd3c84d77 2024-03-11T23:21:58.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: format: 2 2024-03-11T23:21:58.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: features: layering 2024-03-11T23:21:58.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: op_features: 2024-03-11T23:21:58.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: flags: 2024-03-11T23:21:58.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: create_timestamp: Mon Mar 11 23:21:57 2024 2024-03-11T23:21:58.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: access_timestamp: Mon Mar 11 23:21:57 2024 2024-03-11T23:21:58.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: modify_timestamp: Mon Mar 11 23:21:57 2024 2024-03-11T23:21:58.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring state: enabled 2024-03-11T23:21:58.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring mode: snapshot 2024-03-11T23:21:58.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring global id: 77eaba8c-b9e4-4bee-9688-582284cc70e2 2024-03-11T23:21:58.190 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: mirroring primary: true 2024-03-11T23:21:58.193 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ stress_write_image cluster2 mirror image_8 2024-03-11T23:21:58.193 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:21:58.193 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:21:58.193 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_8 2024-03-11T23:21:58.193 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-03-11T23:21:58.194 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local duration=10 2024-03-11T23:21:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set +e 2024-03-11T23:21:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-03-11T23:21:58.391 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:100 IOs, 67067904 bytes 2024-03-11T23:21:58.499 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:200 IOs, 106267648 bytes 2024-03-11T23:21:58.601 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:300 IOs, 132273152 bytes 2024-03-11T23:21:58.719 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:400 IOs, 179814400 bytes 2024-03-11T23:21:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:500 IOs, 203281408 bytes 2024-03-11T23:21:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:600 IOs, 216151040 bytes 2024-03-11T23:21:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:700 IOs, 216783872 bytes 2024-03-11T23:21:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:800 IOs, 267577344 bytes 2024-03-11T23:21:59.272 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:900 IOs, 311826432 bytes 2024-03-11T23:21:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1000 IOs, 379835392 bytes 2024-03-11T23:21:59.563 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1100 IOs, 433191936 bytes 2024-03-11T23:21:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1200 IOs, 445188096 bytes 2024-03-11T23:21:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1300 IOs, 479404032 bytes 2024-03-11T23:21:59.864 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1400 IOs, 536706048 bytes 2024-03-11T23:22:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1500 IOs, 590166016 bytes 2024-03-11T23:22:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1600 IOs, 647488512 bytes 2024-03-11T23:22:00.302 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1700 IOs, 679059456 bytes 2024-03-11T23:22:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1800 IOs, 740455424 bytes 2024-03-11T23:22:00.586 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:1900 IOs, 789017600 bytes 2024-03-11T23:22:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2000 IOs, 811823104 bytes 2024-03-11T23:22:00.851 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2100 IOs, 881345536 bytes 2024-03-11T23:22:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2200 IOs, 969596928 bytes 2024-03-11T23:22:01.171 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2300 IOs, 1026714624 bytes 2024-03-11T23:22:01.274 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2400 IOs, 1063030784 bytes 2024-03-11T23:22:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2500 IOs, 1091170304 bytes 2024-03-11T23:22:01.525 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2600 IOs, 1123170304 bytes 2024-03-11T23:22:01.678 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2700 IOs, 1185650688 bytes 2024-03-11T23:22:01.832 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2800 IOs, 1227760640 bytes 2024-03-11T23:22:01.921 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:2900 IOs, 1262863360 bytes 2024-03-11T23:22:02.024 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3000 IOs, 1263478784 bytes 2024-03-11T23:22:02.154 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3100 IOs, 1302183936 bytes 2024-03-11T23:22:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3200 IOs, 1331310592 bytes 2024-03-11T23:22:02.403 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3300 IOs, 1377191936 bytes 2024-03-11T23:22:02.508 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3400 IOs, 1405242368 bytes 2024-03-11T23:22:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3500 IOs, 1448035328 bytes 2024-03-11T23:22:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3600 IOs, 1502840832 bytes 2024-03-11T23:22:02.892 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3700 IOs, 1553830912 bytes 2024-03-11T23:22:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3800 IOs, 1620469760 bytes 2024-03-11T23:22:03.139 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:3900 IOs, 1632715776 bytes 2024-03-11T23:22:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4000 IOs, 1696122880 bytes 2024-03-11T23:22:03.376 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4100 IOs, 1706249216 bytes 2024-03-11T23:22:03.423 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~281s 2024-03-11T23:22:03.452 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4200 IOs, 1706872832 bytes 2024-03-11T23:22:03.535 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4300 IOs, 1737396224 bytes 2024-03-11T23:22:03.671 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4400 IOs, 1772403712 bytes 2024-03-11T23:22:03.784 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4500 IOs, 1787015168 bytes 2024-03-11T23:22:03.888 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4600 IOs, 1835653120 bytes 2024-03-11T23:22:03.988 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4700 IOs, 1859892224 bytes 2024-03-11T23:22:04.138 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:4800 IOs, 1893150720 bytes 2024-03-11T23:22:04.156 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr:2024-03-11T23:22:04.154+0000 7f4dc6656640 -1 bdev(0x55f17cf76e00 /var/lib/ceph/osd/cluster2-1/block) _aio_thread got r=-28 ((28) No space left on device) 2024-03-11T23:22:04.157 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1979-gb117a47a/rpm/el9/BUILD/ceph-19.0.0-1979-gb117a47a/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f4dc6656640 time 2024-03-11T23:22:04.154701+0000 2024-03-11T23:22:04.157 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1979-gb117a47a/rpm/el9/BUILD/ceph-19.0.0-1979-gb117a47a/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2024-03-11T23:22:04.157 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: ceph version 19.0.0-1979-gb117a47a (b117a47aa0ad81d207fdd9ba852068f2f262fea8) squid (dev) 2024-03-11T23:22:04.157 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x55f1793e64a3] 2024-03-11T23:22:04.157 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 2: (KernelDevice::_aio_thread()+0x11e0) [0x55f179db7550] 2024-03-11T23:22:04.157 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 3: ceph-osd(+0xdc66c1) [0x55f179db76c1] 2024-03-11T23:22:04.157 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 4: /lib64/libc.so.6(+0x9f802) [0x7f4dd3c9f802] 2024-03-11T23:22:04.157 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 5: /lib64/libc.so.6(+0x3f450) [0x7f4dd3c3f450] 2024-03-11T23:22:04.157 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr:2024-03-11T23:22:04.155+0000 7f4dc6656640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1979-gb117a47a/rpm/el9/BUILD/ceph-19.0.0-1979-gb117a47a/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f4dc6656640 time 2024-03-11T23:22:04.154701+0000 2024-03-11T23:22:04.158 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1979-gb117a47a/rpm/el9/BUILD/ceph-19.0.0-1979-gb117a47a/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2024-03-11T23:22:04.158 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 2024-03-11T23:22:04.158 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: ceph version 19.0.0-1979-gb117a47a (b117a47aa0ad81d207fdd9ba852068f2f262fea8) squid (dev) 2024-03-11T23:22:04.158 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x55f1793e64a3] 2024-03-11T23:22:04.158 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 2: (KernelDevice::_aio_thread()+0x11e0) [0x55f179db7550] 2024-03-11T23:22:04.158 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 3: ceph-osd(+0xdc66c1) [0x55f179db76c1] 2024-03-11T23:22:04.158 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 4: /lib64/libc.so.6(+0x9f802) [0x7f4dd3c9f802] 2024-03-11T23:22:04.158 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 5: /lib64/libc.so.6(+0x3f450) [0x7f4dd3c3f450] 2024-03-11T23:22:04.158 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 2024-03-11T23:22:04.158 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr:*** Caught signal (Aborted) ** 2024-03-11T23:22:04.158 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: in thread 7f4dc6656640 thread_name:bstore_aio 2024-03-11T23:22:04.159 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: ceph version 19.0.0-1979-gb117a47a (b117a47aa0ad81d207fdd9ba852068f2f262fea8) squid (dev) 2024-03-11T23:22:04.159 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f4dd3c54db0] 2024-03-11T23:22:04.159 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f4dd3ca154c] 2024-03-11T23:22:04.159 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 3: raise() 2024-03-11T23:22:04.159 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 4: abort() 2024-03-11T23:22:04.159 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x55f1793e6564] 2024-03-11T23:22:04.159 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 6: (KernelDevice::_aio_thread()+0x11e0) [0x55f179db7550] 2024-03-11T23:22:04.159 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 7: ceph-osd(+0xdc66c1) [0x55f179db76c1] 2024-03-11T23:22:04.159 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7f4dd3c9f802] 2024-03-11T23:22:04.159 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7f4dd3c3f450] 2024-03-11T23:22:04.159 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr:2024-03-11T23:22:04.157+0000 7f4dc6656640 -1 *** Caught signal (Aborted) ** 2024-03-11T23:22:04.159 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: in thread 7f4dc6656640 thread_name:bstore_aio 2024-03-11T23:22:04.159 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 2024-03-11T23:22:04.159 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: ceph version 19.0.0-1979-gb117a47a (b117a47aa0ad81d207fdd9ba852068f2f262fea8) squid (dev) 2024-03-11T23:22:04.160 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f4dd3c54db0] 2024-03-11T23:22:04.160 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f4dd3ca154c] 2024-03-11T23:22:04.160 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 3: raise() 2024-03-11T23:22:04.160 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 4: abort() 2024-03-11T23:22:04.160 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x55f1793e6564] 2024-03-11T23:22:04.160 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 6: (KernelDevice::_aio_thread()+0x11e0) [0x55f179db7550] 2024-03-11T23:22:04.160 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 7: ceph-osd(+0xdc66c1) [0x55f179db76c1] 2024-03-11T23:22:04.160 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7f4dd3c9f802] 2024-03-11T23:22:04.160 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7f4dd3c3f450] 2024-03-11T23:22:04.160 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-11T23:22:04.160 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 2024-03-11T23:22:04.172 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: -406> 2024-03-11T23:22:04.154+0000 7f4dc6656640 -1 bdev(0x55f17cf76e00 /var/lib/ceph/osd/cluster2-1/block) _aio_thread got r=-28 ((28) No space left on device) 2024-03-11T23:22:04.173 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: -102> 2024-03-11T23:22:04.155+0000 7f4dc6656640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1979-gb117a47a/rpm/el9/BUILD/ceph-19.0.0-1979-gb117a47a/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f4dc6656640 time 2024-03-11T23:22:04.154701+0000 2024-03-11T23:22:04.173 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1979-gb117a47a/rpm/el9/BUILD/ceph-19.0.0-1979-gb117a47a/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2024-03-11T23:22:04.173 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 2024-03-11T23:22:04.173 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: ceph version 19.0.0-1979-gb117a47a (b117a47aa0ad81d207fdd9ba852068f2f262fea8) squid (dev) 2024-03-11T23:22:04.173 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x55f1793e64a3] 2024-03-11T23:22:04.173 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 2: (KernelDevice::_aio_thread()+0x11e0) [0x55f179db7550] 2024-03-11T23:22:04.173 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 3: ceph-osd(+0xdc66c1) [0x55f179db76c1] 2024-03-11T23:22:04.173 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 4: /lib64/libc.so.6(+0x9f802) [0x7f4dd3c9f802] 2024-03-11T23:22:04.173 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 5: /lib64/libc.so.6(+0x3f450) [0x7f4dd3c3f450] 2024-03-11T23:22:04.173 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 2024-03-11T23:22:04.173 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 0> 2024-03-11T23:22:04.157+0000 7f4dc6656640 -1 *** Caught signal (Aborted) ** 2024-03-11T23:22:04.174 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: in thread 7f4dc6656640 thread_name:bstore_aio 2024-03-11T23:22:04.174 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 2024-03-11T23:22:04.174 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: ceph version 19.0.0-1979-gb117a47a (b117a47aa0ad81d207fdd9ba852068f2f262fea8) squid (dev) 2024-03-11T23:22:04.174 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f4dd3c54db0] 2024-03-11T23:22:04.174 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f4dd3ca154c] 2024-03-11T23:22:04.174 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 3: raise() 2024-03-11T23:22:04.174 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 4: abort() 2024-03-11T23:22:04.174 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x55f1793e6564] 2024-03-11T23:22:04.174 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 6: (KernelDevice::_aio_thread()+0x11e0) [0x55f179db7550] 2024-03-11T23:22:04.174 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 7: ceph-osd(+0xdc66c1) [0x55f179db76c1] 2024-03-11T23:22:04.174 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7f4dd3c9f802] 2024-03-11T23:22:04.174 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7f4dd3c3f450] 2024-03-11T23:22:04.174 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-11T23:22:04.175 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 2024-03-11T23:22:04.181 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: -1407> 2024-03-11T23:22:04.154+0000 7f4dc6656640 -1 bdev(0x55f17cf76e00 /var/lib/ceph/osd/cluster2-1/block) _aio_thread got r=-28 ((28) No space left on device) 2024-03-11T23:22:04.182 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: -1103> 2024-03-11T23:22:04.155+0000 7f4dc6656640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1979-gb117a47a/rpm/el9/BUILD/ceph-19.0.0-1979-gb117a47a/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f4dc6656640 time 2024-03-11T23:22:04.154701+0000 2024-03-11T23:22:04.182 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1979-gb117a47a/rpm/el9/BUILD/ceph-19.0.0-1979-gb117a47a/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2024-03-11T23:22:04.182 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 2024-03-11T23:22:04.182 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: ceph version 19.0.0-1979-gb117a47a (b117a47aa0ad81d207fdd9ba852068f2f262fea8) squid (dev) 2024-03-11T23:22:04.182 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x55f1793e64a3] 2024-03-11T23:22:04.182 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 2: (KernelDevice::_aio_thread()+0x11e0) [0x55f179db7550] 2024-03-11T23:22:04.182 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 3: ceph-osd(+0xdc66c1) [0x55f179db76c1] 2024-03-11T23:22:04.182 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 4: /lib64/libc.so.6(+0x9f802) [0x7f4dd3c9f802] 2024-03-11T23:22:04.182 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 5: /lib64/libc.so.6(+0x3f450) [0x7f4dd3c3f450] 2024-03-11T23:22:04.182 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 2024-03-11T23:22:04.183 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: -1001> 2024-03-11T23:22:04.157+0000 7f4dc6656640 -1 *** Caught signal (Aborted) ** 2024-03-11T23:22:04.183 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: in thread 7f4dc6656640 thread_name:bstore_aio 2024-03-11T23:22:04.183 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 2024-03-11T23:22:04.183 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: ceph version 19.0.0-1979-gb117a47a (b117a47aa0ad81d207fdd9ba852068f2f262fea8) squid (dev) 2024-03-11T23:22:04.183 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f4dd3c54db0] 2024-03-11T23:22:04.183 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f4dd3ca154c] 2024-03-11T23:22:04.183 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 3: raise() 2024-03-11T23:22:04.183 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 4: abort() 2024-03-11T23:22:04.183 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x55f1793e6564] 2024-03-11T23:22:04.183 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 6: (KernelDevice::_aio_thread()+0x11e0) [0x55f179db7550] 2024-03-11T23:22:04.183 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 7: ceph-osd(+0xdc66c1) [0x55f179db76c1] 2024-03-11T23:22:04.183 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7f4dd3c9f802] 2024-03-11T23:22:04.183 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7f4dd3c3f450] 2024-03-11T23:22:04.183 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-11T23:22:04.184 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr: 2024-03-11T23:22:04.396 INFO:tasks.ceph.cluster2.osd.1.smithi132.stderr:daemon-helper: command failed with exit status 5 2024-03-11T23:22:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ error_code=124 2024-03-11T23:22:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ set -e 2024-03-11T23:22:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ '[' 124 -eq 124 ']' 2024-03-11T23:22:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:22:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ wait_for_image_present cluster1 mirror image_8 present 2024-03-11T23:22:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:22:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:22:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_8 2024-03-11T23:22:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local state=present 2024-03-11T23:22:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id= 2024-03-11T23:22:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local s 2024-03-11T23:22:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n '' 2024-03-11T23:22:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_8 2024-03-11T23:22:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:22:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:22:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_8 2024-03-11T23:22:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-03-11T23:22:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:22:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ image_id=12b0fad23e7e 2024-03-11T23:22:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-11T23:22:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sleep 0.1 2024-03-11T23:22:08.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test_image_present cluster1 mirror image_8 present 12b0fad23e7e 2024-03-11T23:22:08.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster1 2024-03-11T23:22:08.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:22:08.350 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_8 2024-03-11T23:22:08.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local test_state=present 2024-03-11T23:22:08.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image_id=12b0fad23e7e 2024-03-11T23:22:08.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_state=deleted 2024-03-11T23:22:08.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local current_image_id 2024-03-11T23:22:08.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ get_image_id cluster1 mirror image_8 2024-03-11T23:22:08.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster=cluster1 2024-03-11T23:22:08.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local pool=mirror 2024-03-11T23:22:08.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local image=image_8 2024-03-11T23:22:08.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-03-11T23:22:08.351 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-11T23:22:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_image_id=12b0fad23e7e 2024-03-11T23:22:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -n 12b0fad23e7e 2024-03-11T23:22:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test -z 12b0fad23e7e -o 12b0fad23e7e = 12b0fad23e7e 2024-03-11T23:22:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ current_state=present 2024-03-11T23:22:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ test present = present 2024-03-11T23:22:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ return 0 2024-03-11T23:22:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ disable_mirror cluster2 mirror image_8 2024-03-11T23:22:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local cluster=cluster2 2024-03-11T23:22:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local pool=mirror 2024-03-11T23:22:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ local image=image_8 2024-03-11T23:22:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_8 2024-03-11T23:22:08.726 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~286s 2024-03-11T23:22:08.726 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~0s 2024-03-11T23:22:14.029 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~292s 2024-03-11T23:22:14.029 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~5s 2024-03-11T23:22:19.333 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~297s 2024-03-11T23:22:19.333 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~11s 2024-03-11T23:22:24.636 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~302s 2024-03-11T23:22:24.637 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~16s 2024-03-11T23:22:24.637 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-03-11T23:22:24.940 INFO:tasks.ceph.cluster2.osd.2:Sent signal 15 2024-03-11T23:22:24.940 INFO:tasks.ceph.cluster2.mon.a:Sent signal 15 2024-03-11T23:22:24.940 INFO:tasks.ceph.cluster2.mgr.x:Sent signal 15 2024-03-11T23:22:24.941 INFO:tasks.ceph.cluster2.osd.2.smithi132.stderr:2024-03-11T23:22:24.939+0000 7f6383df8640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 (PID: 34290) UID: 0 2024-03-11T23:22:24.941 INFO:tasks.ceph.cluster2.osd.2.smithi132.stderr:2024-03-11T23:22:24.939+0000 7f6383df8640 -1 osd.2 371 *** Got signal Terminated *** 2024-03-11T23:22:24.941 INFO:tasks.ceph.cluster2.osd.2.smithi132.stderr:2024-03-11T23:22:24.939+0000 7f6383df8640 -1 osd.2 371 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T23:22:24.942 INFO:tasks.ceph.cluster2.mon.a.smithi132.stderr:2024-03-11T23:22:24.939+0000 7f8da253d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster cluster2 -i a (PID: 33980) UID: 0 2024-03-11T23:22:24.942 INFO:tasks.ceph.cluster2.mon.a.smithi132.stderr:2024-03-11T23:22:24.939+0000 7f8da253d640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-11T23:22:24.943 INFO:tasks.ceph.cluster2.mgr.x.smithi078.stderr:2024-03-11T23:22:24.940+0000 7fa6e4ee1640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster cluster2 -i x (PID: 38734) UID: 0 2024-03-11T23:22:24.943 INFO:tasks.ceph.cluster2.mgr.x.smithi078.stderr:2024-03-11T23:22:24.940+0000 7fa6e4ee1640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-11T23:23:10.977 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9752514346614718) > weight (0.25) 2024-03-11T23:23:10.977 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6534225092173156) > weight (0.25) 2024-03-11T23:23:10.977 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.4249760017631913) > weight (0.25) 2024-03-11T23:23:10.977 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.7770313570035673) > weight (0.25) 2024-03-11T23:23:10.977 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2024-03-11T23:23:24.057 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-03-11T23:23:24.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-03-11T23:23:24.059 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-03-11T23:23:24.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr:2024-03-11T23:23:24.058+0000 7f6def923640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 63080) UID: 1000 2024-03-11T23:23:32.060 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-03-11T23:23:32.060 INFO:teuthology.orchestra.run:waiting for 600 2024-03-11T23:24:25.978 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.38835626565242176) > weight (0.25) 2024-03-11T23:24:25.978 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9435953655289668) > weight (0.25) 2024-03-11T23:24:25.978 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.385508752728809) > weight (0.25) 2024-03-11T23:24:25.979 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.809880212692262) > weight (0.25) 2024-03-11T23:24:25.979 INFO:tasks.rbd_mirror_thrash:waiting for 70 secs before thrashing 2024-03-11T23:25:35.980 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8630288259638406) > weight (0.25) 2024-03-11T23:25:35.980 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-03-11T23:25:35.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-03-11T23:25:35.981 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-03-11T23:25:35.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr:2024-03-11T23:25:35.981+0000 7f9b4d129640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 74049) UID: 1000 2024-03-11T23:25:40.982 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-03-11T23:25:40.982 INFO:teuthology.orchestra.run:waiting for 600 2024-03-11T23:33:36.202 INFO:tasks.rbd_mirror_thrash:Failed to stop cluster2.client.mirror.0 2024-03-11T23:33:36.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 6 2024-03-11T23:33:36.203 ERROR:tasks.rbd_mirror_thrash:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_idryomov_ceph_a4679433626e24f222f64f757c6ab31760167603/qa/tasks/rbd_mirror_thrash.py", line 84, in _run self.do_thrash() File "/home/teuthworker/src/github.com_idryomov_ceph_a4679433626e24f222f64f757c6ab31760167603/qa/tasks/rbd_mirror_thrash.py", line 156, in do_thrash run.wait([daemon.proc], timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (101) after waiting for 600 seconds 2024-03-11T23:33:36.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr:*** Caught signal (Aborted) ** 2024-03-11T23:33:36.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: in thread 7f6df2e21640 thread_name:rbd-mirror 2024-03-11T23:33:36.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: ceph version 19.0.0-1979-gb117a47a (b117a47aa0ad81d207fdd9ba852068f2f262fea8) squid (dev) 2024-03-11T23:33:36.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f6df2c54db0] 2024-03-11T23:33:36.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7f6df2c9c39a] 2024-03-11T23:33:36.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 3: pthread_cond_wait() 2024-03-11T23:33:36.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f6df30d5500] 2024-03-11T23:33:36.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 5: /lib64/librados.so.2(+0xa49ff) [0x7f6df49ad9ff] 2024-03-11T23:33:36.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 6: (librados::v14_2_0::IoCtx::operate(std::__cxx11::basic_string, std::allocator > const&, librados::v14_2_0::ObjectReadOperation*, ceph::buffer::v15_2_0::list*)+0x65) [0x7f6df498a6e5] 2024-03-11T23:33:36.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 7: (librbd::cls_client::mirror_mode_get(librados::v14_2_0::IoCtx*, cls::rbd::MirrorMode*)+0xa8) [0x555be3b2e2d8] 2024-03-11T23:33:36.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 8: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x45d) [0x555be35ebe1d] 2024-03-11T23:33:36.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 9: (rbd::mirror::ClusterWatcher::refresh_pools()+0x78) [0x555be35ed878] 2024-03-11T23:33:36.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 10: (rbd::mirror::Mirror::run()+0xb4) [0x555be35ccb94] 2024-03-11T23:33:36.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 11: main() 2024-03-11T23:33:36.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 12: /lib64/libc.so.6(+0x3feb0) [0x7f6df2c3feb0] 2024-03-11T23:33:36.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 13: __libc_start_main() 2024-03-11T23:33:36.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 14: _start() 2024-03-11T23:33:36.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr:2024-03-11T23:33:36.205+0000 7f6df2e21640 -1 *** Caught signal (Aborted) ** 2024-03-11T23:33:36.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: in thread 7f6df2e21640 thread_name:rbd-mirror 2024-03-11T23:33:36.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 2024-03-11T23:33:36.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: ceph version 19.0.0-1979-gb117a47a (b117a47aa0ad81d207fdd9ba852068f2f262fea8) squid (dev) 2024-03-11T23:33:36.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f6df2c54db0] 2024-03-11T23:33:36.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7f6df2c9c39a] 2024-03-11T23:33:36.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 3: pthread_cond_wait() 2024-03-11T23:33:36.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f6df30d5500] 2024-03-11T23:33:36.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 5: /lib64/librados.so.2(+0xa49ff) [0x7f6df49ad9ff] 2024-03-11T23:33:36.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 6: (librados::v14_2_0::IoCtx::operate(std::__cxx11::basic_string, std::allocator > const&, librados::v14_2_0::ObjectReadOperation*, ceph::buffer::v15_2_0::list*)+0x65) [0x7f6df498a6e5] 2024-03-11T23:33:36.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 7: (librbd::cls_client::mirror_mode_get(librados::v14_2_0::IoCtx*, cls::rbd::MirrorMode*)+0xa8) [0x555be3b2e2d8] 2024-03-11T23:33:36.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 8: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x45d) [0x555be35ebe1d] 2024-03-11T23:33:36.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 9: (rbd::mirror::ClusterWatcher::refresh_pools()+0x78) [0x555be35ed878] 2024-03-11T23:33:36.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 10: (rbd::mirror::Mirror::run()+0xb4) [0x555be35ccb94] 2024-03-11T23:33:36.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 11: main() 2024-03-11T23:33:36.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 12: /lib64/libc.so.6(+0x3feb0) [0x7f6df2c3feb0] 2024-03-11T23:33:36.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 13: __libc_start_main() 2024-03-11T23:33:36.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 14: _start() 2024-03-11T23:33:36.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-11T23:33:36.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 2024-03-11T23:33:36.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -8531> 2024-03-11T23:14:37.234+0000 7f6de8915640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555be6096b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -8528> 2024-03-11T23:14:37.234+0000 7f6dee921640 -1 rbd::mirror::ImageReplayer: 0x555be6818c80 [3/0db5ab54-08c1-4a9b-b8f1-d2ac90c55bee] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -8378> 2024-03-11T23:14:50.522+0000 7f6de8915640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555be5fa0ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -8375> 2024-03-11T23:14:50.522+0000 7f6dee921640 -1 rbd::mirror::ImageReplayer: 0x555be68ca000 [3/b02ac2fb-bf75-42d9-bf3f-a8404d4fe797] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -8276> 2024-03-11T23:14:52.536+0000 7f6de8114640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555be67e0d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -8273> 2024-03-11T23:14:52.536+0000 7f6def122640 -1 rbd::mirror::ImageReplayer: 0x555be6818780 [3/6e7e7d8e-c0d0-4968-90fa-7731ffa367f1] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -8126> 2024-03-11T23:15:04.500+0000 7f6de8114640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555be67e11e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -8123> 2024-03-11T23:15:04.500+0000 7f6def122640 -1 rbd::mirror::ImageReplayer: 0x555be6819b80 [3/8afe84ee-3bfc-43a7-b04d-7925c8799653] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -7115> 2024-03-11T23:15:08.608+0000 7f6de8915640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555be5fa0000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -7112> 2024-03-11T23:15:08.608+0000 7f6dee921640 -1 rbd::mirror::ImageReplayer: 0x555be6818000 [3/387adbd8-be3f-401f-ba8e-7062c5a54dfd] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -6914> 2024-03-11T23:15:30.545+0000 7f6de8915640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555be5fa0b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -6911> 2024-03-11T23:15:30.545+0000 7f6dee921640 -1 rbd::mirror::ImageReplayer: 0x555be6819400 [3/56839585-8992-435c-80a3-a4d4b2cb652c] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -6041> 2024-03-11T23:15:44.504+0000 7f6de8114640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555be5fa2ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -6038> 2024-03-11T23:15:44.504+0000 7f6def122640 -1 rbd::mirror::ImageReplayer: 0x555be6818f00 [3/0ecf1da7-76ab-4437-ad6e-4ee82f47a299] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -5913> 2024-03-11T23:15:52.605+0000 7f6de8114640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555be5fa3380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -5910> 2024-03-11T23:15:52.605+0000 7f6def122640 -1 rbd::mirror::ImageReplayer: 0x555be6819900 [3/5c2e35fd-9ddc-477b-abb2-86d6a62016b3] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -5785> 2024-03-11T23:16:00.657+0000 7f6de8915640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555be5fa31e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -5782> 2024-03-11T23:16:00.657+0000 7f6def122640 -1 rbd::mirror::ImageReplayer: 0x555be6819680 [3/5772827a-fff4-4729-bef7-03d63872ad43] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -5685> 2024-03-11T23:16:02.668+0000 7f6de8114640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555be6097040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -5682> 2024-03-11T23:16:02.668+0000 7f6def122640 -1 rbd::mirror::ImageReplayer: 0x555be6818a00 [3/d3a33f3e-a3e4-4e65-b3f9-eb5141260b1a] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -5250> 2024-03-11T23:16:10.706+0000 7f6de8915640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555be6099a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -5247> 2024-03-11T23:16:10.706+0000 7f6dee921640 -1 rbd::mirror::ImageReplayer: 0x555be6818500 [3/71ab5b7e-440a-48ab-8f45-bb970a72ca7e] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -5150> 2024-03-11T23:16:12.708+0000 7f6de8114640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555be6099ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -5147> 2024-03-11T23:16:12.708+0000 7f6dee921640 -1 rbd::mirror::ImageReplayer: 0x555be6818280 [3/3fb7dde7-de15-40cd-9bc8-e3db123cf169] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -1714> 2024-03-11T23:23:24.058+0000 7f6def923640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 63080) UID: 1000 2024-03-11T23:33:36.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 0> 2024-03-11T23:33:36.205+0000 7f6df2e21640 -1 *** Caught signal (Aborted) ** 2024-03-11T23:33:36.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: in thread 7f6df2e21640 thread_name:rbd-mirror 2024-03-11T23:33:36.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 2024-03-11T23:33:36.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: ceph version 19.0.0-1979-gb117a47a (b117a47aa0ad81d207fdd9ba852068f2f262fea8) squid (dev) 2024-03-11T23:33:36.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f6df2c54db0] 2024-03-11T23:33:36.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7f6df2c9c39a] 2024-03-11T23:33:36.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 3: pthread_cond_wait() 2024-03-11T23:33:36.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f6df30d5500] 2024-03-11T23:33:36.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 5: /lib64/librados.so.2(+0xa49ff) [0x7f6df49ad9ff] 2024-03-11T23:33:36.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 6: (librados::v14_2_0::IoCtx::operate(std::__cxx11::basic_string, std::allocator > const&, librados::v14_2_0::ObjectReadOperation*, ceph::buffer::v15_2_0::list*)+0x65) [0x7f6df498a6e5] 2024-03-11T23:33:36.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 7: (librbd::cls_client::mirror_mode_get(librados::v14_2_0::IoCtx*, cls::rbd::MirrorMode*)+0xa8) [0x555be3b2e2d8] 2024-03-11T23:33:36.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 8: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x45d) [0x555be35ebe1d] 2024-03-11T23:33:36.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 9: (rbd::mirror::ClusterWatcher::refresh_pools()+0x78) [0x555be35ed878] 2024-03-11T23:33:36.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 10: (rbd::mirror::Mirror::run()+0xb4) [0x555be35ccb94] 2024-03-11T23:33:36.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 11: main() 2024-03-11T23:33:36.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 12: /lib64/libc.so.6(+0x3feb0) [0x7f6df2c3feb0] 2024-03-11T23:33:36.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 13: __libc_start_main() 2024-03-11T23:33:36.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 14: _start() 2024-03-11T23:33:36.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-11T23:33:36.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 2024-03-11T23:33:36.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -9999> 2024-03-11T23:14:37.234+0000 7f6de8915640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555be6096b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -9998> 2024-03-11T23:14:37.234+0000 7f6dee921640 -1 rbd::mirror::ImageReplayer: 0x555be6818c80 [3/0db5ab54-08c1-4a9b-b8f1-d2ac90c55bee] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -9997> 2024-03-11T23:14:50.522+0000 7f6de8915640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555be5fa0ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -9996> 2024-03-11T23:14:50.522+0000 7f6dee921640 -1 rbd::mirror::ImageReplayer: 0x555be68ca000 [3/b02ac2fb-bf75-42d9-bf3f-a8404d4fe797] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -9995> 2024-03-11T23:14:52.536+0000 7f6de8114640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555be67e0d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -9994> 2024-03-11T23:14:52.536+0000 7f6def122640 -1 rbd::mirror::ImageReplayer: 0x555be6818780 [3/6e7e7d8e-c0d0-4968-90fa-7731ffa367f1] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -9993> 2024-03-11T23:15:04.500+0000 7f6de8114640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555be67e11e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -9992> 2024-03-11T23:15:04.500+0000 7f6def122640 -1 rbd::mirror::ImageReplayer: 0x555be6819b80 [3/8afe84ee-3bfc-43a7-b04d-7925c8799653] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -9991> 2024-03-11T23:15:08.608+0000 7f6de8915640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555be5fa0000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -9990> 2024-03-11T23:15:08.608+0000 7f6dee921640 -1 rbd::mirror::ImageReplayer: 0x555be6818000 [3/387adbd8-be3f-401f-ba8e-7062c5a54dfd] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -9989> 2024-03-11T23:15:30.545+0000 7f6de8915640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555be5fa0b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -9988> 2024-03-11T23:15:30.545+0000 7f6dee921640 -1 rbd::mirror::ImageReplayer: 0x555be6819400 [3/56839585-8992-435c-80a3-a4d4b2cb652c] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -9987> 2024-03-11T23:15:44.504+0000 7f6de8114640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555be5fa2ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -9986> 2024-03-11T23:15:44.504+0000 7f6def122640 -1 rbd::mirror::ImageReplayer: 0x555be6818f00 [3/0ecf1da7-76ab-4437-ad6e-4ee82f47a299] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -9985> 2024-03-11T23:15:52.605+0000 7f6de8114640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555be5fa3380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -9984> 2024-03-11T23:15:52.605+0000 7f6def122640 -1 rbd::mirror::ImageReplayer: 0x555be6819900 [3/5c2e35fd-9ddc-477b-abb2-86d6a62016b3] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -9983> 2024-03-11T23:16:00.657+0000 7f6de8915640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555be5fa31e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -9982> 2024-03-11T23:16:00.657+0000 7f6def122640 -1 rbd::mirror::ImageReplayer: 0x555be6819680 [3/5772827a-fff4-4729-bef7-03d63872ad43] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -9981> 2024-03-11T23:16:02.668+0000 7f6de8114640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555be6097040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -9980> 2024-03-11T23:16:02.668+0000 7f6def122640 -1 rbd::mirror::ImageReplayer: 0x555be6818a00 [3/d3a33f3e-a3e4-4e65-b3f9-eb5141260b1a] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -9979> 2024-03-11T23:16:10.706+0000 7f6de8915640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555be6099a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -9978> 2024-03-11T23:16:10.706+0000 7f6dee921640 -1 rbd::mirror::ImageReplayer: 0x555be6818500 [3/71ab5b7e-440a-48ab-8f45-bb970a72ca7e] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -9977> 2024-03-11T23:16:12.708+0000 7f6de8114640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555be6099ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -9976> 2024-03-11T23:16:12.708+0000 7f6dee921640 -1 rbd::mirror::ImageReplayer: 0x555be6818280 [3/3fb7dde7-de15-40cd-9bc8-e3db123cf169] operator(): start failed: (85) Interrupted system call should be restarted 2024-03-11T23:33:36.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -9975> 2024-03-11T23:23:24.058+0000 7f6def923640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 63080) UID: 1000 2024-03-11T23:33:36.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: -9974> 2024-03-11T23:33:36.205+0000 7f6df2e21640 -1 *** Caught signal (Aborted) ** 2024-03-11T23:33:36.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: in thread 7f6df2e21640 thread_name:rbd-mirror 2024-03-11T23:33:36.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 2024-03-11T23:33:36.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: ceph version 19.0.0-1979-gb117a47a (b117a47aa0ad81d207fdd9ba852068f2f262fea8) squid (dev) 2024-03-11T23:33:36.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f6df2c54db0] 2024-03-11T23:33:36.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7f6df2c9c39a] 2024-03-11T23:33:36.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 3: pthread_cond_wait() 2024-03-11T23:33:36.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f6df30d5500] 2024-03-11T23:33:36.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 5: /lib64/librados.so.2(+0xa49ff) [0x7f6df49ad9ff] 2024-03-11T23:33:36.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 6: (librados::v14_2_0::IoCtx::operate(std::__cxx11::basic_string, std::allocator > const&, librados::v14_2_0::ObjectReadOperation*, ceph::buffer::v15_2_0::list*)+0x65) [0x7f6df498a6e5] 2024-03-11T23:33:36.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 7: (librbd::cls_client::mirror_mode_get(librados::v14_2_0::IoCtx*, cls::rbd::MirrorMode*)+0xa8) [0x555be3b2e2d8] 2024-03-11T23:33:36.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 8: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x45d) [0x555be35ebe1d] 2024-03-11T23:33:36.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 9: (rbd::mirror::ClusterWatcher::refresh_pools()+0x78) [0x555be35ed878] 2024-03-11T23:33:36.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 10: (rbd::mirror::Mirror::run()+0xb4) [0x555be35ccb94] 2024-03-11T23:33:36.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 11: main() 2024-03-11T23:33:36.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 12: /lib64/libc.so.6(+0x3feb0) [0x7f6df2c3feb0] 2024-03-11T23:33:36.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 13: __libc_start_main() 2024-03-11T23:33:36.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 14: _start() 2024-03-11T23:33:36.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-11T23:33:36.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr: 2024-03-11T23:33:36.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi132.stderr:daemon-helper: command crashed with signal 6 2024-03-11T23:35:45.123 INFO:tasks.rbd_mirror_thrash:Failed to stop cluster1.client.mirror.1 2024-03-11T23:35:45.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 6 2024-03-11T23:35:45.124 ERROR:tasks.rbd_mirror_thrash:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_idryomov_ceph_a4679433626e24f222f64f757c6ab31760167603/qa/tasks/rbd_mirror_thrash.py", line 84, in _run self.do_thrash() File "/home/teuthworker/src/github.com_idryomov_ceph_a4679433626e24f222f64f757c6ab31760167603/qa/tasks/rbd_mirror_thrash.py", line 156, in do_thrash run.wait([daemon.proc], timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (101) after waiting for 600 seconds 2024-03-11T23:35:45.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr:*** Caught signal (Aborted) ** 2024-03-11T23:35:45.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: in thread 7f9b5017f640 thread_name:rbd-mirror 2024-03-11T23:35:45.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: ceph version 19.0.0-1979-gb117a47a (b117a47aa0ad81d207fdd9ba852068f2f262fea8) squid (dev) 2024-03-11T23:35:45.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f9b50454db0] 2024-03-11T23:35:45.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7f9b5049c39a] 2024-03-11T23:35:45.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 3: pthread_cond_wait() 2024-03-11T23:35:45.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f9b508d5500] 2024-03-11T23:35:45.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 5: rbd-mirror(+0x36cadb) [0x55ebcbb4fadb] 2024-03-11T23:35:45.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 6: (rbd::mirror::PoolReplayer::shut_down()+0x103) [0x55ebcbb60043] 2024-03-11T23:35:45.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 7: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x55ebcbb53370] 2024-03-11T23:35:45.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 8: rbd-mirror(+0x98b318) [0x55ebcc16e318] 2024-03-11T23:35:45.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 9: rbd-mirror(+0x98b300) [0x55ebcc16e300] 2024-03-11T23:35:45.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 10: (rbd::mirror::Mirror::~Mirror()+0xca) [0x55ebcbb5189a] 2024-03-11T23:35:45.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 11: main() 2024-03-11T23:35:45.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 12: /lib64/libc.so.6(+0x3feb0) [0x7f9b5043feb0] 2024-03-11T23:35:45.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 13: __libc_start_main() 2024-03-11T23:35:45.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 14: _start() 2024-03-11T23:35:45.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr:2024-03-11T23:35:45.127+0000 7f9b5017f640 -1 *** Caught signal (Aborted) ** 2024-03-11T23:35:45.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: in thread 7f9b5017f640 thread_name:rbd-mirror 2024-03-11T23:35:45.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 2024-03-11T23:35:45.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: ceph version 19.0.0-1979-gb117a47a (b117a47aa0ad81d207fdd9ba852068f2f262fea8) squid (dev) 2024-03-11T23:35:45.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f9b50454db0] 2024-03-11T23:35:45.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7f9b5049c39a] 2024-03-11T23:35:45.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 3: pthread_cond_wait() 2024-03-11T23:35:45.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f9b508d5500] 2024-03-11T23:35:45.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 5: rbd-mirror(+0x36cadb) [0x55ebcbb4fadb] 2024-03-11T23:35:45.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 6: (rbd::mirror::PoolReplayer::shut_down()+0x103) [0x55ebcbb60043] 2024-03-11T23:35:45.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 7: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x55ebcbb53370] 2024-03-11T23:35:45.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 8: rbd-mirror(+0x98b318) [0x55ebcc16e318] 2024-03-11T23:35:45.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 9: rbd-mirror(+0x98b300) [0x55ebcc16e300] 2024-03-11T23:35:45.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 10: (rbd::mirror::Mirror::~Mirror()+0xca) [0x55ebcbb5189a] 2024-03-11T23:35:45.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 11: main() 2024-03-11T23:35:45.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 12: /lib64/libc.so.6(+0x3feb0) [0x7f9b5043feb0] 2024-03-11T23:35:45.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 13: __libc_start_main() 2024-03-11T23:35:45.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 14: _start() 2024-03-11T23:35:45.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-11T23:35:45.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 2024-03-11T23:35:45.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: -2790> 2024-03-11T23:25:35.981+0000 7f9b4d129640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 74049) UID: 1000 2024-03-11T23:35:45.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 0> 2024-03-11T23:35:45.127+0000 7f9b5017f640 -1 *** Caught signal (Aborted) ** 2024-03-11T23:35:45.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: in thread 7f9b5017f640 thread_name:rbd-mirror 2024-03-11T23:35:45.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 2024-03-11T23:35:45.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: ceph version 19.0.0-1979-gb117a47a (b117a47aa0ad81d207fdd9ba852068f2f262fea8) squid (dev) 2024-03-11T23:35:45.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f9b50454db0] 2024-03-11T23:35:45.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7f9b5049c39a] 2024-03-11T23:35:45.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 3: pthread_cond_wait() 2024-03-11T23:35:45.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f9b508d5500] 2024-03-11T23:35:45.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 5: rbd-mirror(+0x36cadb) [0x55ebcbb4fadb] 2024-03-11T23:35:45.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 6: (rbd::mirror::PoolReplayer::shut_down()+0x103) [0x55ebcbb60043] 2024-03-11T23:35:45.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 7: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x55ebcbb53370] 2024-03-11T23:35:45.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 8: rbd-mirror(+0x98b318) [0x55ebcc16e318] 2024-03-11T23:35:45.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 9: rbd-mirror(+0x98b300) [0x55ebcc16e300] 2024-03-11T23:35:45.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 10: (rbd::mirror::Mirror::~Mirror()+0xca) [0x55ebcbb5189a] 2024-03-11T23:35:45.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 11: main() 2024-03-11T23:35:45.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 12: /lib64/libc.so.6(+0x3feb0) [0x7f9b5043feb0] 2024-03-11T23:35:45.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 13: __libc_start_main() 2024-03-11T23:35:45.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 14: _start() 2024-03-11T23:35:45.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-11T23:35:45.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 2024-03-11T23:35:45.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: -7634> 2024-03-11T23:25:35.981+0000 7f9b4d129640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 74049) UID: 1000 2024-03-11T23:35:45.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: -7633> 2024-03-11T23:35:45.127+0000 7f9b5017f640 -1 *** Caught signal (Aborted) ** 2024-03-11T23:35:45.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: in thread 7f9b5017f640 thread_name:rbd-mirror 2024-03-11T23:35:45.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 2024-03-11T23:35:45.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: ceph version 19.0.0-1979-gb117a47a (b117a47aa0ad81d207fdd9ba852068f2f262fea8) squid (dev) 2024-03-11T23:35:45.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f9b50454db0] 2024-03-11T23:35:45.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7f9b5049c39a] 2024-03-11T23:35:45.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 3: pthread_cond_wait() 2024-03-11T23:35:45.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f9b508d5500] 2024-03-11T23:35:45.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 5: rbd-mirror(+0x36cadb) [0x55ebcbb4fadb] 2024-03-11T23:35:45.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 6: (rbd::mirror::PoolReplayer::shut_down()+0x103) [0x55ebcbb60043] 2024-03-11T23:35:45.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 7: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x55ebcbb53370] 2024-03-11T23:35:45.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 8: rbd-mirror(+0x98b318) [0x55ebcc16e318] 2024-03-11T23:35:45.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 9: rbd-mirror(+0x98b300) [0x55ebcc16e300] 2024-03-11T23:35:45.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 10: (rbd::mirror::Mirror::~Mirror()+0xca) [0x55ebcbb5189a] 2024-03-11T23:35:45.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 11: main() 2024-03-11T23:35:45.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 12: /lib64/libc.so.6(+0x3feb0) [0x7f9b5043feb0] 2024-03-11T23:35:45.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 13: __libc_start_main() 2024-03-11T23:35:45.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 14: _start() 2024-03-11T23:35:45.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-11T23:35:45.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr: 2024-03-11T23:35:45.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi132.stderr:daemon-helper: command crashed with signal 6 2024-03-11T23:35:45.849 INFO:tasks.daemonwatchdog.daemon_watchdog:thrasher.rbd_mirror.[cluster1] failed 2024-03-11T23:35:45.849 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-03-11T23:35:46.354 INFO:tasks.ceph.cluster1.osd.0:Sent signal 15 2024-03-11T23:35:46.354 INFO:tasks.ceph.cluster1.osd.1:Sent signal 15 2024-03-11T23:35:46.354 INFO:tasks.ceph.cluster1.osd.2:Sent signal 15 2024-03-11T23:35:46.354 INFO:tasks.ceph.cluster1.mon.a:Sent signal 15 2024-03-11T23:35:46.354 INFO:tasks.ceph.cluster1.mgr.x:Sent signal 15 2024-03-11T23:35:46.355 INFO:tasks.ceph.cluster1.osd.0.smithi078.stderr:2024-03-11T23:35:46.356+0000 7ff7b8289640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 (PID: 33353) UID: 0 2024-03-11T23:35:46.355 INFO:tasks.ceph.cluster1.osd.0.smithi078.stderr:2024-03-11T23:35:46.356+0000 7ff7b8289640 -1 osd.0 444 *** Got signal Terminated *** 2024-03-11T23:35:46.355 INFO:tasks.ceph.cluster1.osd.0.smithi078.stderr:2024-03-11T23:35:46.356+0000 7ff7b8289640 -1 osd.0 444 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T23:35:46.356 INFO:tasks.ceph.cluster1.osd.1.smithi078.stderr:2024-03-11T23:35:46.356+0000 7f0d96501640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 (PID: 33356) UID: 0 2024-03-11T23:35:46.356 INFO:tasks.ceph.cluster1.osd.1.smithi078.stderr:2024-03-11T23:35:46.356+0000 7f0d96501640 -1 osd.1 444 *** Got signal Terminated *** 2024-03-11T23:35:46.356 INFO:tasks.ceph.cluster1.osd.1.smithi078.stderr:2024-03-11T23:35:46.356+0000 7f0d96501640 -1 osd.1 444 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T23:35:46.356 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2024-03-11T23:35:46.356+0000 7ff0ce917640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 (PID: 33358) UID: 0 2024-03-11T23:35:46.356 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2024-03-11T23:35:46.356+0000 7ff0ce917640 -1 osd.2 444 *** Got signal Terminated *** 2024-03-11T23:35:46.356 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2024-03-11T23:35:46.356+0000 7ff0ce917640 -1 osd.2 444 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T23:35:46.356 INFO:tasks.ceph.cluster1.mon.a.smithi078.stderr:2024-03-11T23:35:46.357+0000 7f890f21c640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster cluster1 -i a (PID: 33021) UID: 0 2024-03-11T23:35:46.357 INFO:tasks.ceph.cluster1.mon.a.smithi078.stderr:2024-03-11T23:35:46.357+0000 7f890f21c640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-11T23:35:46.357 INFO:tasks.ceph.cluster1.mgr.x.smithi078.stderr:2024-03-11T23:35:46.357+0000 7f17943e8640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster cluster1 -i x (PID: 33023) UID: 0 2024-03-11T23:35:46.357 INFO:tasks.ceph.cluster1.mgr.x.smithi078.stderr:2024-03-11T23:35:46.357+0000 7f17943e8640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-12T04:49:06.576 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:Terminated 2024-03-12T04:49:06.605 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ cleanup 143 2024-03-12T04:49:06.605 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster1 status 2024-03-12T04:49:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local error_code=143 2024-03-12T04:49:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ set +e 2024-03-12T04:49:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ '[' 143 -ne 0 ']' 2024-03-12T04:49:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ status 2024-03-12T04:49:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ local cluster daemon image_pool image_ns image 2024-03-12T04:49:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for cluster in ${CLUSTER1} ${CLUSTER2} 2024-03-12T04:49:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster1 status' 2024-03-12T04:49:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-12T04:49:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --cluster cluster1 -s 2024-03-12T04:54:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:2024-03-12T04:54:06.654+0000 7feb9c497640 0 monclient(hunting): authenticate timed out after 300 2024-03-12T04:54:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-03-12T04:54:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-12T04:54:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --cluster cluster1 service dump 2024-03-12T04:59:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:2024-03-12T04:59:06.738+0000 7f6e9d69a640 0 monclient(hunting): authenticate timed out after 300 2024-03-12T04:59:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-03-12T04:59:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-12T04:59:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --cluster cluster1 service status 2024-03-12T05:04:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:2024-03-12T05:04:06.822+0000 7f54062d1640 0 monclient(hunting): authenticate timed out after 300 2024-03-12T05:04:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-03-12T05:04:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T05:04:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T05:04:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for image_pool in ${POOL} ${PARENT_POOL} 2024-03-12T05:04:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for image_ns in "" "${NS1}" "${NS2}" 2024-03-12T05:04:06.826 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster1 mirror images 2024-03-12T05:04:06.826 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster1 mirror images' 2024-03-12T05:04:06.826 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 -p mirror --namespace '' ls -l 2024-03-12T05:09:06.856 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: couldn't connect to the cluster! 2024-03-12T05:09:06.856 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: listing images failed: (110) Connection timed out 2024-03-12T05:09:06.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T05:09:06.857 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T05:09:06.858 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster1 mirror mirror pool info 2024-03-12T05:09:06.859 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster1 mirror mirror pool info' 2024-03-12T05:09:06.859 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 -p mirror --namespace '' mirror pool info 2024-03-12T05:14:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: couldn't connect to the cluster! 2024-03-12T05:14:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T05:14:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T05:14:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster1 mirror mirror pool status' 2024-03-12T05:14:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster1 mirror mirror pool status 2024-03-12T05:14:06.914 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-12T05:14:06.914 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 -p mirror --namespace '' mirror pool status --verbose 2024-03-12T05:19:06.918 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: couldn't connect to the cluster! 2024-03-12T05:19:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T05:19:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T05:19:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ rbd --cluster cluster1 -p mirror --namespace '' ls 2024-03-12T05:24:06.952 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster1 mirror rbd_mirroring omap vals 2024-03-12T05:24:06.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster1 mirror rbd_mirroring omap vals' 2024-03-12T05:24:06.953 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rados --cluster cluster1 -p mirror --namespace '' listomapvals rbd_mirroring 2024-03-12T05:29:06.979 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-03-12T05:29:06.980 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster1 mirror rbd_mirror_leader omap vals 2024-03-12T05:29:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster1 mirror rbd_mirror_leader omap vals' 2024-03-12T05:29:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rados --cluster cluster1 -p mirror --namespace '' listomapvals rbd_mirror_leader 2024-03-12T05:34:07.008 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-03-12T05:34:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T05:34:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T05:34:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for image_ns in "" "${NS1}" "${NS2}" 2024-03-12T05:34:07.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster1 mirror ns1 images 2024-03-12T05:34:07.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster1 mirror ns1 images' 2024-03-12T05:34:07.010 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 -p mirror --namespace ns1 ls -l 2024-03-12T05:39:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: couldn't connect to the cluster! 2024-03-12T05:39:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: listing images failed: (110) Connection timed out 2024-03-12T05:39:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T05:39:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T05:39:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster1 mirrorns1 mirror pool info 2024-03-12T05:39:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster1 mirrorns1 mirror pool info' 2024-03-12T05:39:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 -p mirror --namespace ns1 mirror pool info 2024-03-12T05:44:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: couldn't connect to the cluster! 2024-03-12T05:44:07.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T05:44:07.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T05:44:07.112 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster1 mirrorns1 mirror pool status 2024-03-12T05:44:07.113 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster1 mirrorns1 mirror pool status' 2024-03-12T05:44:07.113 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-12T05:44:07.113 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 -p mirror --namespace ns1 mirror pool status --verbose 2024-03-12T05:49:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: couldn't connect to the cluster! 2024-03-12T05:49:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T05:49:07.116 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T05:49:07.116 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ rbd --cluster cluster1 -p mirror --namespace ns1 ls 2024-03-12T05:54:07.149 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster1 mirror ns1 rbd_mirroring omap vals 2024-03-12T05:54:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster1 mirror ns1 rbd_mirroring omap vals' 2024-03-12T05:54:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rados --cluster cluster1 -p mirror --namespace ns1 listomapvals rbd_mirroring 2024-03-12T05:59:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-03-12T05:59:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster1 mirror ns1 rbd_mirror_leader omap vals' 2024-03-12T05:59:07.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster1 mirror ns1 rbd_mirror_leader omap vals 2024-03-12T05:59:07.179 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rados --cluster cluster1 -p mirror --namespace ns1 listomapvals rbd_mirror_leader 2024-03-12T06:04:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-03-12T06:04:07.208 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T06:04:07.208 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T06:04:07.208 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for image_ns in "" "${NS1}" "${NS2}" 2024-03-12T06:04:07.209 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster1 mirror ns2 images 2024-03-12T06:04:07.209 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster1 mirror ns2 images' 2024-03-12T06:04:07.209 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 -p mirror --namespace ns2 ls -l 2024-03-12T06:09:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: couldn't connect to the cluster! 2024-03-12T06:09:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: listing images failed: (110) Connection timed out 2024-03-12T06:09:07.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T06:09:07.240 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T06:09:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster1 mirrorns2 mirror pool info 2024-03-12T06:09:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster1 mirrorns2 mirror pool info' 2024-03-12T06:09:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 -p mirror --namespace ns2 mirror pool info 2024-03-12T06:14:07.271 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: couldn't connect to the cluster! 2024-03-12T06:14:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T06:14:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T06:14:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster1 mirrorns2 mirror pool status' 2024-03-12T06:14:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster1 mirrorns2 mirror pool status 2024-03-12T06:14:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-12T06:14:07.274 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 -p mirror --namespace ns2 mirror pool status --verbose 2024-03-12T06:19:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: couldn't connect to the cluster! 2024-03-12T06:19:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T06:19:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T06:19:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ rbd --cluster cluster1 -p mirror --namespace ns2 ls 2024-03-12T06:24:07.337 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster1 mirror ns2 rbd_mirroring omap vals 2024-03-12T06:24:07.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster1 mirror ns2 rbd_mirroring omap vals' 2024-03-12T06:24:07.341 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rados --cluster cluster1 -p mirror --namespace ns2 listomapvals rbd_mirroring 2024-03-12T06:29:07.367 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-03-12T06:29:07.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster1 mirror ns2 rbd_mirror_leader omap vals 2024-03-12T06:29:07.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster1 mirror ns2 rbd_mirror_leader omap vals' 2024-03-12T06:29:07.368 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rados --cluster cluster1 -p mirror --namespace ns2 listomapvals rbd_mirror_leader 2024-03-12T06:34:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-03-12T06:34:07.396 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T06:34:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T06:34:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for image_pool in ${POOL} ${PARENT_POOL} 2024-03-12T06:34:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for image_ns in "" "${NS1}" "${NS2}" 2024-03-12T06:34:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster1 mirror_parent images' 2024-03-12T06:34:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster1 mirror_parent images 2024-03-12T06:34:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace '' ls -l 2024-03-12T06:39:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: couldn't connect to the cluster! 2024-03-12T06:39:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: listing images failed: (110) Connection timed out 2024-03-12T06:39:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T06:39:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T06:39:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster1 mirror_parent mirror pool info 2024-03-12T06:39:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster1 mirror_parent mirror pool info' 2024-03-12T06:39:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace '' mirror pool info 2024-03-12T06:44:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: couldn't connect to the cluster! 2024-03-12T06:44:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T06:44:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T06:44:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster1 mirror_parent mirror pool status' 2024-03-12T06:44:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster1 mirror_parent mirror pool status 2024-03-12T06:44:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-12T06:44:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace '' mirror pool status --verbose 2024-03-12T06:49:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: couldn't connect to the cluster! 2024-03-12T06:49:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T06:49:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T06:49:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ rbd --cluster cluster1 -p mirror_parent --namespace '' ls 2024-03-12T06:54:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster1 mirror_parent rbd_mirroring omap vals 2024-03-12T06:54:07.556 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster1 mirror_parent rbd_mirroring omap vals' 2024-03-12T06:54:07.556 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rados --cluster cluster1 -p mirror_parent --namespace '' listomapvals rbd_mirroring 2024-03-12T06:59:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-03-12T06:59:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster1 mirror_parent rbd_mirror_leader omap vals 2024-03-12T06:59:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster1 mirror_parent rbd_mirror_leader omap vals' 2024-03-12T06:59:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rados --cluster cluster1 -p mirror_parent --namespace '' listomapvals rbd_mirror_leader 2024-03-12T07:04:07.581 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-03-12T07:04:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T07:04:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T07:04:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for image_ns in "" "${NS1}" "${NS2}" 2024-03-12T07:04:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster1 mirror_parent ns1 images' 2024-03-12T07:04:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster1 mirror_parent ns1 images 2024-03-12T07:04:07.584 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace ns1 ls -l 2024-03-12T07:09:07.612 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: couldn't connect to the cluster! 2024-03-12T07:09:07.613 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: listing images failed: (110) Connection timed out 2024-03-12T07:09:07.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T07:09:07.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T07:09:07.614 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster1 mirror_parentns1 mirror pool info' 2024-03-12T07:09:07.615 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster1 mirror_parentns1 mirror pool info 2024-03-12T07:09:07.615 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace ns1 mirror pool info 2024-03-12T07:14:07.646 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: couldn't connect to the cluster! 2024-03-12T07:14:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T07:14:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T07:14:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster1 mirror_parentns1 mirror pool status' 2024-03-12T07:14:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster1 mirror_parentns1 mirror pool status 2024-03-12T07:14:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-12T07:14:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace ns1 mirror pool status --verbose 2024-03-12T07:19:07.679 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: couldn't connect to the cluster! 2024-03-12T07:19:07.680 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T07:19:07.681 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T07:19:07.681 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ rbd --cluster cluster1 -p mirror_parent --namespace ns1 ls 2024-03-12T07:24:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster1 mirror_parent ns1 rbd_mirroring omap vals 2024-03-12T07:24:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster1 mirror_parent ns1 rbd_mirroring omap vals' 2024-03-12T07:24:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rados --cluster cluster1 -p mirror_parent --namespace ns1 listomapvals rbd_mirroring 2024-03-12T07:29:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-03-12T07:29:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster1 mirror_parent ns1 rbd_mirror_leader omap vals 2024-03-12T07:29:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster1 mirror_parent ns1 rbd_mirror_leader omap vals' 2024-03-12T07:29:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rados --cluster cluster1 -p mirror_parent --namespace ns1 listomapvals rbd_mirror_leader 2024-03-12T07:34:07.768 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-03-12T07:34:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T07:34:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T07:34:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for image_ns in "" "${NS1}" "${NS2}" 2024-03-12T07:34:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster1 mirror_parent ns2 images 2024-03-12T07:34:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster1 mirror_parent ns2 images' 2024-03-12T07:34:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace ns2 ls -l 2024-03-12T07:39:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: couldn't connect to the cluster! 2024-03-12T07:39:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: listing images failed: (110) Connection timed out 2024-03-12T07:39:07.800 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T07:39:07.800 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T07:39:07.801 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster1 mirror_parentns2 mirror pool info 2024-03-12T07:39:07.801 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster1 mirror_parentns2 mirror pool info' 2024-03-12T07:39:07.801 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace ns2 mirror pool info 2024-03-12T07:44:07.834 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: couldn't connect to the cluster! 2024-03-12T07:44:07.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T07:44:07.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T07:44:07.836 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster1 mirror_parentns2 mirror pool status 2024-03-12T07:44:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster1 mirror_parentns2 mirror pool status' 2024-03-12T07:44:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-12T07:44:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace ns2 mirror pool status --verbose 2024-03-12T07:49:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: couldn't connect to the cluster! 2024-03-12T07:49:07.868 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T07:49:07.869 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T07:49:07.869 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ rbd --cluster cluster1 -p mirror_parent --namespace ns2 ls 2024-03-12T07:54:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster1 mirror_parent ns2 rbd_mirroring omap vals 2024-03-12T07:54:07.902 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster1 mirror_parent ns2 rbd_mirroring omap vals' 2024-03-12T07:54:07.902 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rados --cluster cluster1 -p mirror_parent --namespace ns2 listomapvals rbd_mirroring 2024-03-12T07:59:07.930 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-03-12T07:59:07.931 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster1 mirror_parent ns2 rbd_mirror_leader omap vals 2024-03-12T07:59:07.931 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster1 mirror_parent ns2 rbd_mirror_leader omap vals' 2024-03-12T07:59:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rados --cluster cluster1 -p mirror_parent --namespace ns2 listomapvals rbd_mirror_leader 2024-03-12T08:04:07.959 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-03-12T08:04:07.960 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T08:04:07.960 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T08:04:07.960 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for cluster in ${CLUSTER1} ${CLUSTER2} 2024-03-12T08:04:07.960 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster2 status 2024-03-12T08:04:07.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster2 status' 2024-03-12T08:04:07.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-12T08:04:07.961 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --cluster cluster2 -s 2024-03-12T08:09:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:2024-03-12T08:09:08.039+0000 7f2ea0153640 0 monclient(hunting): authenticate timed out after 300 2024-03-12T08:09:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-03-12T08:09:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-12T08:09:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --cluster cluster2 service dump 2024-03-12T08:14:08.126 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:2024-03-12T08:14:08.123+0000 7f6c94881640 0 monclient(hunting): authenticate timed out after 300 2024-03-12T08:14:08.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-03-12T08:14:08.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-12T08:14:08.127 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ ceph --cluster cluster2 service status 2024-03-12T08:19:08.214 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:2024-03-12T08:19:08.210+0000 7f73b8bf1640 0 monclient(hunting): authenticate timed out after 300 2024-03-12T08:19:08.214 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-03-12T08:19:08.215 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T08:19:08.215 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T08:19:08.215 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for image_pool in ${POOL} ${PARENT_POOL} 2024-03-12T08:19:08.215 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for image_ns in "" "${NS1}" "${NS2}" 2024-03-12T08:19:08.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster2 mirror images 2024-03-12T08:19:08.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster2 mirror images' 2024-03-12T08:19:08.216 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster2 -p mirror --namespace '' ls -l 2024-03-12T08:24:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: couldn't connect to the cluster! 2024-03-12T08:24:08.246 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: listing images failed: (110) Connection timed out 2024-03-12T08:24:08.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T08:24:08.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T08:24:08.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster2 mirror mirror pool info 2024-03-12T08:24:08.247 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster2 mirror mirror pool info' 2024-03-12T08:24:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster2 -p mirror --namespace '' mirror pool info 2024-03-12T08:29:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: couldn't connect to the cluster! 2024-03-12T08:29:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T08:29:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T08:29:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster2 mirror mirror pool status 2024-03-12T08:29:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster2 mirror mirror pool status' 2024-03-12T08:29:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-12T08:29:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster2 -p mirror --namespace '' mirror pool status --verbose 2024-03-12T08:34:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: couldn't connect to the cluster! 2024-03-12T08:34:08.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T08:34:08.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T08:34:08.315 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ rbd --cluster cluster2 -p mirror --namespace '' ls 2024-03-12T08:39:08.348 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster2 mirror rbd_mirroring omap vals 2024-03-12T08:39:08.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster2 mirror rbd_mirroring omap vals' 2024-03-12T08:39:08.349 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rados --cluster cluster2 -p mirror --namespace '' listomapvals rbd_mirroring 2024-03-12T08:44:08.377 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-03-12T08:44:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster2 mirror rbd_mirror_leader omap vals 2024-03-12T08:44:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster2 mirror rbd_mirror_leader omap vals' 2024-03-12T08:44:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rados --cluster cluster2 -p mirror --namespace '' listomapvals rbd_mirror_leader 2024-03-12T08:49:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-03-12T08:49:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T08:49:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T08:49:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for image_ns in "" "${NS1}" "${NS2}" 2024-03-12T08:49:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster2 mirror ns1 images 2024-03-12T08:49:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster2 mirror ns1 images' 2024-03-12T08:49:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster2 -p mirror --namespace ns1 ls -l 2024-03-12T08:54:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: couldn't connect to the cluster! 2024-03-12T08:54:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: listing images failed: (110) Connection timed out 2024-03-12T08:54:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T08:54:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T08:54:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster2 mirrorns1 mirror pool info' 2024-03-12T08:54:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster2 mirrorns1 mirror pool info 2024-03-12T08:54:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster2 -p mirror --namespace ns1 mirror pool info 2024-03-12T08:59:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: couldn't connect to the cluster! 2024-03-12T08:59:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T08:59:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster2 mirrorns1 mirror pool status 2024-03-12T08:59:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T08:59:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster2 mirrorns1 mirror pool status' 2024-03-12T08:59:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-12T08:59:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster2 -p mirror --namespace ns1 mirror pool status --verbose 2024-03-12T09:04:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: couldn't connect to the cluster! 2024-03-12T09:04:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T09:04:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T09:04:08.505 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ rbd --cluster cluster2 -p mirror --namespace ns1 ls 2024-03-12T09:09:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster2 mirror ns1 rbd_mirroring omap vals 2024-03-12T09:09:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster2 mirror ns1 rbd_mirroring omap vals' 2024-03-12T09:09:08.538 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rados --cluster cluster2 -p mirror --namespace ns1 listomapvals rbd_mirroring 2024-03-12T09:14:08.565 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-03-12T09:14:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster2 mirror ns1 rbd_mirror_leader omap vals 2024-03-12T09:14:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster2 mirror ns1 rbd_mirror_leader omap vals' 2024-03-12T09:14:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rados --cluster cluster2 -p mirror --namespace ns1 listomapvals rbd_mirror_leader 2024-03-12T09:19:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-03-12T09:19:08.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T09:19:08.595 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T09:19:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for image_ns in "" "${NS1}" "${NS2}" 2024-03-12T09:19:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster2 mirror ns2 images' 2024-03-12T09:19:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster2 mirror ns2 images 2024-03-12T09:19:08.597 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster2 -p mirror --namespace ns2 ls -l 2024-03-12T09:24:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: couldn't connect to the cluster! 2024-03-12T09:24:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: listing images failed: (110) Connection timed out 2024-03-12T09:24:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T09:24:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T09:24:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster2 mirrorns2 mirror pool info 2024-03-12T09:24:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster2 mirrorns2 mirror pool info' 2024-03-12T09:24:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster2 -p mirror --namespace ns2 mirror pool info 2024-03-12T09:29:08.658 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: couldn't connect to the cluster! 2024-03-12T09:29:08.660 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T09:29:08.660 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T09:29:08.660 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster2 mirrorns2 mirror pool status 2024-03-12T09:29:08.660 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster2 mirrorns2 mirror pool status' 2024-03-12T09:29:08.661 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-12T09:29:08.661 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster2 -p mirror --namespace ns2 mirror pool status --verbose 2024-03-12T09:34:08.691 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: couldn't connect to the cluster! 2024-03-12T09:34:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T09:34:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T09:34:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ rbd --cluster cluster2 -p mirror --namespace ns2 ls 2024-03-12T09:39:08.725 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster2 mirror ns2 rbd_mirroring omap vals 2024-03-12T09:39:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster2 mirror ns2 rbd_mirroring omap vals' 2024-03-12T09:39:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rados --cluster cluster2 -p mirror --namespace ns2 listomapvals rbd_mirroring 2024-03-12T09:44:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-03-12T09:44:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster2 mirror ns2 rbd_mirror_leader omap vals 2024-03-12T09:44:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster2 mirror ns2 rbd_mirror_leader omap vals' 2024-03-12T09:44:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rados --cluster cluster2 -p mirror --namespace ns2 listomapvals rbd_mirror_leader 2024-03-12T09:49:08.780 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-03-12T09:49:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T09:49:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T09:49:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for image_pool in ${POOL} ${PARENT_POOL} 2024-03-12T09:49:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for image_ns in "" "${NS1}" "${NS2}" 2024-03-12T09:49:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster2 mirror_parent images 2024-03-12T09:49:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster2 mirror_parent images' 2024-03-12T09:49:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace '' ls -l 2024-03-12T09:54:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: couldn't connect to the cluster! 2024-03-12T09:54:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: listing images failed: (110) Connection timed out 2024-03-12T09:54:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T09:54:08.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T09:54:08.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster2 mirror_parent mirror pool info 2024-03-12T09:54:08.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster2 mirror_parent mirror pool info' 2024-03-12T09:54:08.813 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace '' mirror pool info 2024-03-12T09:59:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: couldn't connect to the cluster! 2024-03-12T09:59:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T09:59:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T09:59:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster2 mirror_parent mirror pool status 2024-03-12T09:59:08.847 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster2 mirror_parent mirror pool status' 2024-03-12T09:59:08.847 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-12T09:59:08.847 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace '' mirror pool status --verbose 2024-03-12T10:04:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: couldn't connect to the cluster! 2024-03-12T10:04:08.879 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T10:04:08.879 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T10:04:08.879 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ rbd --cluster cluster2 -p mirror_parent --namespace '' ls 2024-03-12T10:09:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster2 mirror_parent rbd_mirroring omap vals 2024-03-12T10:09:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster2 mirror_parent rbd_mirroring omap vals' 2024-03-12T10:09:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rados --cluster cluster2 -p mirror_parent --namespace '' listomapvals rbd_mirroring 2024-03-12T10:14:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-03-12T10:14:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster2 mirror_parent rbd_mirror_leader omap vals 2024-03-12T10:14:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster2 mirror_parent rbd_mirror_leader omap vals' 2024-03-12T10:14:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rados --cluster cluster2 -p mirror_parent --namespace '' listomapvals rbd_mirror_leader 2024-03-12T10:19:08.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-03-12T10:19:08.968 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T10:19:08.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T10:19:08.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ for image_ns in "" "${NS1}" "${NS2}" 2024-03-12T10:19:08.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster2 mirror_parent ns1 images 2024-03-12T10:19:08.969 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster2 mirror_parent ns1 images' 2024-03-12T10:19:08.970 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace ns1 ls -l 2024-03-12T10:24:08.998 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: couldn't connect to the cluster! 2024-03-12T10:24:08.999 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: listing images failed: (110) Connection timed out 2024-03-12T10:24:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T10:24:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T10:24:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster2 mirror_parentns1 mirror pool info 2024-03-12T10:24:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster2 mirror_parentns1 mirror pool info' 2024-03-12T10:24:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace ns1 mirror pool info 2024-03-12T10:29:09.032 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: couldn't connect to the cluster! 2024-03-12T10:29:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T10:29:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T10:29:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:cluster2 mirror_parentns1 mirror pool status 2024-03-12T10:29:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 'cluster2 mirror_parentns1 mirror pool status' 2024-03-12T10:29:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ CEPH_ARGS= 2024-03-12T10:29:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace ns1 mirror pool status --verbose 2024-03-12T10:34:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:rbd: couldn't connect to the cluster! 2024-03-12T10:34:09.068 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout: 2024-03-12T10:34:09.068 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:++ echo 2024-03-12T10:34:09.068 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+++ rbd --cluster cluster2 -p mirror_parent --namespace ns1 ls 2024-03-12T10:37:47.147 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2024-03-12T10:37:47.148 DEBUG:teuthology.task.console_log:Killing console logger for smithi078 2024-03-12T10:37:47.149 DEBUG:teuthology.task.console_log:Killing console logger for smithi132 2024-03-12T10:37:47.149 DEBUG:teuthology.exit:Finished running handlers